Jan 12 03:36:01 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3145" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jan 13 02:02:26 mds1 init: tty (/dev/tty1) main process (3851) killed by TERM signal
Jan 13 02:02:26 mds1 init: tty (/dev/tty2) main process (3853) killed by TERM signal
Jan 13 02:02:26 mds1 init: tty (/dev/tty3) main process (3855) killed by TERM signal
Jan 13 02:02:26 mds1 init: tty (/dev/tty4) main process (3857) killed by TERM signal
Jan 13 02:02:26 mds1 init: tty (/dev/tty5) main process (3859) killed by TERM signal
Jan 13 02:02:26 mds1 init: tty (/dev/tty6) main process (3861) killed by TERM signal
Jan 13 02:02:26 mds1 abrtd: Got signal 15, exiting
Jan 13 02:02:30 mds1 acpid: exiting
Jan 13 02:02:30 mds1 multipathd: mdt2: event checker exit
Jan 13 02:02:30 mds1 multipathd: dm-1: remove map (uevent)
Jan 13 02:02:30 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 13 02:02:30 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 13 02:02:30 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 13 02:02:30 mds1 multipathd: dm-1: remove map (uevent)
Jan 13 02:02:30 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 13 02:02:30 mds1 multipathd: mgt: stop event checker thread (139762503878400)
Jan 13 02:02:30 mds1 multipathd: mdt1: stop event checker thread (139762493101824)
Jan 13 02:02:30 mds1 multipathd: --------shut down-------
Jan 13 02:02:30 mds1 init: Disconnected from system bus
Jan 13 02:02:30 mds1 console-kit-daemon[3760]: WARNING: no sender#012
Jan 13 02:02:31 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 13 02:02:31 mds1 auditd[3120]: The audit daemon is exiting.
Jan 13 02:02:31 mds1 kernel: type=1305 audit(1389596551.190:1320): audit_pid=0 old=3120 auid=4294967295 ses=4294967295 res=1
Jan 13 02:05:31 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 02:05:31 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3112" x-info="http://www.rsyslog.com"] start
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 02:05:31 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 02:05:31 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 02:05:31 mds1 kernel: KERNEL supported cpus:
Jan 13 02:05:31 mds1 kernel:  Intel GenuineIntel
Jan 13 02:05:31 mds1 kernel:  AMD AuthenticAMD
Jan 13 02:05:31 mds1 kernel:  Centaur CentaurHauls
Jan 13 02:05:31 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 02:05:31 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 02:05:31 mds1 kernel: DMI 2.6 present.
Jan 13 02:05:31 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 02:05:31 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 02:05:31 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 02:05:31 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 02:05:31 mds1 kernel: Using GB pages for direct mapping
Jan 13 02:05:31 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 02:05:31 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 02:05:31 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 02:05:31 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 02:05:31 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 02:05:31 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 02:05:31 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:05:31 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:05:31 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:05:31 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:05:31 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:05:31 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 02:05:31 mds1 kernel: Setting APIC routing to flat.
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 02:05:31 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 02:05:31 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 02:05:31 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 02:05:31 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 02:05:31 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 02:05:31 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 02:05:31 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 02:05:31 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 02:05:31 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 02:05:31 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 02:05:31 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 02:05:31 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 02:05:31 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 02:05:31 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 02:05:31 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 02:05:31 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 02:05:31 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 02:05:31 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 02:05:31 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 02:05:31 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 02:05:31 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 02:05:31 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 02:05:31 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 02:05:31 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 02:05:31 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 02:05:31 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 02:05:31 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 02:05:31 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 02:05:31 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 02:05:31 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 02:05:31 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 02:05:31 mds1 kernel: Zone PFN ranges:
Jan 13 02:05:31 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 02:05:31 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 02:05:31 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 02:05:31 mds1 kernel: Movable zone start PFN for each node
Jan 13 02:05:31 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 02:05:31 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 02:05:31 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 02:05:31 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 02:05:31 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 02:05:31 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 02:05:31 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 02:05:31 mds1 kernel: Setting APIC routing to flat.
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 02:05:31 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 02:05:31 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 02:05:31 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 02:05:31 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 02:05:31 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 02:05:31 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 02:05:31 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 02:05:31 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 02:05:31 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 02:05:31 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 02:05:31 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 02:05:31 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 02:05:31 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 02:05:31 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 02:05:31 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 02:05:31 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 02:05:31 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 02:05:31 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 02:05:31 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 02:05:31 mds1 kernel: Policy zone: Normal
Jan 13 02:05:31 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 02:05:31 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 02:05:31 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 02:05:31 mds1 kernel: Checking aperture...
Jan 13 02:05:31 mds1 kernel: No AGP bridge found
Jan 13 02:05:31 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 02:05:31 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 02:05:31 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 02:05:31 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 02:05:31 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 02:05:31 mds1 kernel: Hierarchical RCU implementation.
Jan 13 02:05:31 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 02:05:31 mds1 kernel: Extended CMOS year: 2000
Jan 13 02:05:31 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 02:05:31 mds1 kernel: console [tty0] enabled
Jan 13 02:05:31 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 02:05:31 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 02:05:31 mds1 kernel: Fast TSC calibration using PIT
Jan 13 02:05:31 mds1 kernel: Detected 1995.145 MHz processor.
Jan 13 02:05:31 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.29 BogoMIPS (lpj=1995145)
Jan 13 02:05:31 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 02:05:31 mds1 kernel: Security Framework initialized
Jan 13 02:05:31 mds1 kernel: SELinux:  Initializing.
Jan 13 02:05:31 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 02:05:31 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 02:05:31 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys ns
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys memory
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys devices
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 02:05:31 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 02:05:31 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 02:05:31 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 02:05:31 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 02:05:31 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 02:05:31 mds1 kernel: using mwait in idle threads.
Jan 13 02:05:31 mds1 kernel: ACPI: Core revision 20090903
Jan 13 02:05:31 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 02:05:31 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 02:05:31 mds1 kernel: dmar: Host address width 46
Jan 13 02:05:31 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 02:05:31 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 02:05:31 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 02:05:31 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 02:05:31 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 02:05:31 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 02:05:31 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 02:05:31 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 02:05:31 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 02:05:31 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 02:05:31 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 02:05:31 mds1 kernel: Enabling x2apic
Jan 13 02:05:31 mds1 kernel: Enabled x2apic
Jan 13 02:05:31 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 02:05:31 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 02:05:31 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 02:05:31 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 02:05:31 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 02:05:31 mds1 kernel: ... version:                3
Jan 13 02:05:31 mds1 kernel: ... bit width:              48
Jan 13 02:05:31 mds1 kernel: ... generic registers:      8
Jan 13 02:05:31 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 02:05:31 mds1 kernel: ... max period:             000000007fffffff
Jan 13 02:05:31 mds1 kernel: ... fixed-purpose events:   3
Jan 13 02:05:31 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 02:05:31 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 02:05:31 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 02:05:31 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 02:05:31 mds1 kernel: Brought up 12 CPUs
Jan 13 02:05:31 mds1 kernel: Total of 12 processors activated (47880.81 BogoMIPS).
Jan 13 02:05:31 mds1 kernel: devtmpfs: initialized
Jan 13 02:05:31 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 02:05:31 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 02:05:31 mds1 kernel: regulator: core version 0.5
Jan 13 02:05:31 mds1 kernel: NET: Registered protocol family 16
Jan 13 02:05:31 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 02:05:31 mds1 kernel: ACPI: bus type pci registered
Jan 13 02:05:31 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 02:05:31 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 02:05:31 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 02:05:31 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 02:05:31 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 02:05:31 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 02:05:31 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 02:05:31 mds1 kernel: ACPI: Interpreter enabled
Jan 13 02:05:31 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 02:05:31 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 02:05:31 mds1 kernel: ACPI: No dock devices found.
Jan 13 02:05:31 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 02:05:31 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 02:05:31 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 02:05:31 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 02:05:31 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 02:05:31 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:05:31 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 02:05:31 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 02:05:31 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 02:05:31 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 02:05:31 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 02:05:31 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:05:31 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:05:31 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 02:05:31 mds1 kernel: vgaarb: loaded
Jan 13 02:05:31 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 02:05:31 mds1 kernel: SCSI subsystem initialized
Jan 13 02:05:31 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 02:05:31 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 02:05:31 mds1 kernel: usbcore: registered new device driver usb
Jan 13 02:05:31 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 02:05:31 mds1 kernel: NetLabel: Initializing
Jan 13 02:05:31 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 02:05:31 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 02:05:31 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 02:05:31 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 02:05:31 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 02:05:31 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 02:05:31 mds1 kernel: Switching to clocksource hpet
Jan 13 02:05:31 mds1 kernel: pnp: PnP ACPI init
Jan 13 02:05:31 mds1 kernel: ACPI: bus type pnp registered
Jan 13 02:05:31 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 02:05:31 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 02:05:31 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 02:05:31 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:05:31 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:05:31 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:05:31 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:05:31 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 02:05:31 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:05:31 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:05:31 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:05:31 mds1 kernel: NET: Registered protocol family 2
Jan 13 02:05:31 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 02:05:31 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 02:05:31 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 02:05:31 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 02:05:31 mds1 kernel: TCP reno registered
Jan 13 02:05:31 mds1 kernel: NET: Registered protocol family 1
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 02:05:31 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 02:05:31 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 02:05:31 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 02:05:31 mds1 kernel: dmar: Unsupported device scope
Jan 13 02:05:31 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 02:05:31 mds1 kernel: type=2000 audit(1389596708.056:1): initialized
Jan 13 02:05:31 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 02:05:31 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 02:05:31 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 02:05:31 mds1 kernel: msgmni has been set to 32768
Jan 13 02:05:31 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 02:05:31 mds1 kernel: ksign: Installing public key data
Jan 13 02:05:31 mds1 kernel: Loading keyring
Jan 13 02:05:31 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 02:05:31 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 02:05:31 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 02:05:31 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 02:05:31 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 02:05:31 mds1 kernel: io scheduler noop registered
Jan 13 02:05:31 mds1 kernel: io scheduler anticipatory registered
Jan 13 02:05:31 mds1 kernel: io scheduler deadline registered (default)
Jan 13 02:05:31 mds1 kernel: io scheduler cfq registered
Jan 13 02:05:31 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 02:05:31 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 02:05:31 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 02:05:31 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 02:05:31 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 02:05:31 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 02:05:31 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 02:05:31 mds1 kernel: ipmi message handler version 39.2
Jan 13 02:05:31 mds1 kernel: IPMI System Interface driver.
Jan 13 02:05:31 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 02:05:31 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 02:05:31 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 02:05:31 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 02:05:31 mds1 kernel: IPMI kcs interface initialized
Jan 13 02:05:31 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 02:05:31 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 02:05:31 mds1 kernel: ERST: Could not register with persistent store
Jan 13 02:05:31 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 02:05:31 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 02:05:31 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 02:05:31 mds1 kernel: Linux agpgart interface v0.103
Jan 13 02:05:31 mds1 kernel: crash memory driver: version 1.1
Jan 13 02:05:31 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 02:05:31 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 02:05:31 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 02:05:31 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 02:05:31 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 02:05:31 mds1 kernel: brd: module loaded
Jan 13 02:05:31 mds1 kernel: loop: module loaded
Jan 13 02:05:31 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 02:05:31 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 02:05:31 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 02:05:31 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 02:05:31 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 02:05:31 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 02:05:31 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 02:05:31 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 02:05:31 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 02:05:31 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 02:05:31 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 02:05:31 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 02:05:31 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 02:05:31 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 02:05:31 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 02:05:31 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 02:05:31 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 02:05:31 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 02:05:31 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 02:05:31 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 02:05:31 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 02:05:31 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 02:05:31 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 02:05:31 mds1 kernel: Switching to clocksource tsc
Jan 13 02:05:31 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 02:05:31 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 02:05:31 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 02:05:31 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 02:05:31 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 02:05:31 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 02:05:31 mds1 kernel: cpuidle: using governor ladder
Jan 13 02:05:31 mds1 kernel: cpuidle: using governor menu
Jan 13 02:05:31 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 02:05:31 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 02:05:31 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 02:05:31 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 02:05:31 mds1 kernel: TCP cubic registered
Jan 13 02:05:31 mds1 kernel: Initializing XFRM netlink socket
Jan 13 02:05:31 mds1 kernel: NET: Registered protocol family 17
Jan 13 02:05:31 mds1 kernel: registered taskstats version 1
Jan 13 02:05:31 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 07:05:10 UTC (1389596710)
Jan 13 02:05:31 mds1 kernel: Initalizing network drop monitor service
Jan 13 02:05:31 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 02:05:31 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 02:05:31 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 02:05:31 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 02:05:31 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 02:05:31 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 02:05:31 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 02:05:31 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 02:05:31 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 02:05:31 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 02:05:31 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 02:05:31 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 02:05:31 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 02:05:31 mds1 kernel: udev: starting version 147
Jan 13 02:05:31 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 02:05:31 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 02:05:31 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 02:05:31 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 02:05:31 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 02:05:31 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 02:05:31 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 02:05:31 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 02:05:31 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 02:05:31 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 02:05:31 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 02:05:31 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 02:05:31 mds1 kernel: megasas: FW now in Ready state
Jan 13 02:05:31 mds1 kernel: megasas:IOC Init cmd success
Jan 13 02:05:31 mds1 kernel: megasas: INIT adapter done
Jan 13 02:05:31 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 02:05:31 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 02:05:31 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 02:05:31 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 02:05:31 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 02:05:31 mds1 kernel: scsi1 : qla2xxx
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 02:05:31 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 02:05:31 mds1 kernel: scsi2 : qla2xxx
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 02:05:31 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 02:05:31 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 02:05:31 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 02:05:31 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 02:05:31 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 02:05:31 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 02:05:31 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 02:05:31 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 02:05:31 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2d96, mb[3]=a100, cmd=6a ****.
Jan 13 02:05:31 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2db3, mb[3]=3100, cmd=6a ****.
Jan 13 02:05:31 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:05:31 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 02:05:31 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 02:05:31 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 02:05:31 mds1 kernel: scsi3 : ahci
Jan 13 02:05:31 mds1 kernel: scsi4 : ahci
Jan 13 02:05:31 mds1 kernel: scsi5 : ahci
Jan 13 02:05:31 mds1 kernel: scsi6 : ahci
Jan 13 02:05:31 mds1 kernel: scsi7 : ahci
Jan 13 02:05:31 mds1 kernel: scsi8 : ahci
Jan 13 02:05:31 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 02:05:31 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 02:05:31 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 02:05:31 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 02:05:31 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 02:05:31 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 02:05:31 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 02:05:31 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 02:05:31 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 02:05:31 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 02:05:31 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 02:05:31 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 02:05:31 mds1 kernel: sda:
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:05:31 mds1 kernel: sdc:
Jan 13 02:05:31 mds1 kernel: sdb:
Jan 13 02:05:31 mds1 kernel: sdd: unknown partition table
Jan 13 02:05:31 mds1 kernel: sdf: unknown partition table
Jan 13 02:05:31 mds1 kernel: sde: unknown partition table
Jan 13 02:05:31 mds1 kernel: unknown partition table
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sdg:
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: unknown partition table
Jan 13 02:05:31 mds1 kernel: unknown partition table
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sda1 sda2 sda3
Jan 13 02:05:31 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 02:05:31 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 02:05:31 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 02:05:31 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 02:05:31 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 02:05:31 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 02:05:31 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 02:05:31 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 02:05:31 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 02:05:31 mds1 kernel: type=1404 audit(1389596714.718:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 02:05:31 mds1 kernel: dracut: 
Jan 13 02:05:31 mds1 kernel: dracut: Switching root
Jan 13 02:05:31 mds1 kernel: udev: starting version 147
Jan 13 02:05:31 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 02:05:31 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 02:05:31 mds1 kernel: PTP clock support registered
Jan 13 02:05:31 mds1 kernel: dca service started, version 1.12.1
Jan 13 02:05:31 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 02:05:31 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:05:31 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 02:05:31 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 02:05:31 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 02:05:31 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:05:31 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:05:31 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 02:05:31 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 02:05:31 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 02:05:31 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 02:05:31 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 02:05:31 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 02:05:31 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:05:31 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 02:05:31 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 02:05:31 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 02:05:31 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:05:31 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:05:31 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 02:05:31 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:05:31 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 02:05:31 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 02:05:31 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 02:05:31 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 02:05:31 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:05:31 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 02:05:31 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:05:31 mds1 kernel: emc: device handler registered
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 02:05:31 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 02:05:31 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 02:05:31 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 02:05:31 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 02:05:31 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 02:05:31 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 02:05:31 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 02:05:31 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1026270000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 02:05:31 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1026270000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 02:05:31 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 02:05:31 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 02:05:31 mds1 kernel: NET: Registered protocol family 10
Jan 13 02:05:31 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 02:05:31 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 02:05:31 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 02:05:31 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 02:05:31 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 02:05:31 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:05:31 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 02:05:31 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:05:31 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:05:31 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 02:05:31 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 02:05:31 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 02:05:31 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 02:05:31 mds1 rpc.statd[3207]: Version 1.2.3 starting
Jan 13 02:05:31 mds1 sm-notify[3208]: Version 1.2.3 starting
Jan 13 02:05:31 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 02:05:31 mds1 kernel: RPC: Registered udp transport module.
Jan 13 02:05:31 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 02:05:31 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 02:05:31 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 02:05:31 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 02:05:31 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 02:05:31 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 02:05:35 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 02:05:35 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 02:05:40 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 02:05:40 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 02:05:40 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 02:05:40 mds1 kernel: LNet: Accept secure, port 988
Jan 13 02:05:40 mds1 cib[3370]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:40 mds1 attrd[3373]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:40 mds1 stonith-ng[3371]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:40 mds1 lrmd[3372]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:40 mds1 pengine[3374]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:40 mds1 crmd[3375]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:05:41 mds1 logger[3384]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 02:05:42 mds1 kdump: kexec: loaded kdump kernel
Jan 13 02:05:42 mds1 kdump: started up
Jan 13 02:05:43 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 02:05:43 mds1 kernel: Slow work thread pool: Starting up
Jan 13 02:05:43 mds1 kernel: Slow work thread pool: Ready
Jan 13 02:05:43 mds1 kernel: FS-Cache: Loaded
Jan 13 02:05:43 mds1 kernel: Registering the id_resolver key type
Jan 13 02:05:43 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 02:05:43 mds1 acpid: starting up
Jan 13 02:05:43 mds1 acpid: 1 rule loaded
Jan 13 02:05:43 mds1 acpid: waiting for events: event logging is off
Jan 13 02:05:44 mds1 acpid: client connected from 3600[68:68]
Jan 13 02:05:44 mds1 acpid: 1 client rule loaded
Jan 13 02:05:44 mds1 kernel: ipmi device interface
Jan 13 02:05:45 mds1 automount[3639]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 02:05:46 mds1 abrtd: Init complete, entering main loop
Jan 13 02:05:46 mds1 cimserver[3790]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 02:05:52 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 02:05:54 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 02:06:10 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 02:06:15 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 02:06:15 mds1 kernel: LustreError: 4079:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 13 02:06:15 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 02:06:15 mds1 kernel: LustreError: 4264:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 02:06:15 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:06:20 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389596775/real 0]  req@ffff880fef6b8c00 x1457097791242320/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389596780 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:06:21 mds1 kernel: LustreError: 4079:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8810043a9800 x1457097791242412/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:21 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880827660000 x1457097791242416/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:27 mds1 kernel: LustreError: 4079:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8810043a9800 x1457097791242420/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:27 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 02:06:33 mds1 kernel: LustreError: 4079:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8810043a9800 x1457097791242428/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:33 mds1 kernel: LustreError: 4079:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 02:06:33 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 02:06:39 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880829d3ec00 x1457097791242432/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:39 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 02:06:45 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880829d3ec00 x1457097791242444/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:45 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 02:06:57 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880829d3ec00 x1457097791242456/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:06:57 mds1 kernel: LustreError: 4133:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Jan 13 02:06:58 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:06:58 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:07:05 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 02:07:05 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 02:07:23 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:07:48 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:07:48 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:08:13 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:08:38 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:08:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:09:03 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:09:28 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:09:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:09:53 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 02:10:18 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:11:08 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:11:08 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:11:08 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 02:11:58 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:12:48 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:13:38 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:13:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:13:38 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 02:15:18 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:15:18 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 02:17:48 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:17:48 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 02:18:13 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:18:13 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 13 02:22:48 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:22:48 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 02:23:35 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 02:23:35 mds1 kernel: 
Jan 13 02:24:18 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 13 02:24:18 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 02:24:18 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 02:24:18 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 0@lo when existing nid 192.168.1.42@o2ib is already connected
Jan 13 02:24:18 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 02:24:18 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-MDT0001_UUID seen on new nid 0@lo when existing nid 192.168.1.42@o2ib is already connected
Jan 13 02:24:18 mds1 kernel: Lustre: emc-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 13 02:24:23 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389597858/real 0]  req@ffff88082368f800 x1457097791263352/t0(0) o8->emc-OST0005-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389597863 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:24:23 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 02:24:28 mds1 kernel: Lustre: 4442:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389597858/real 1389597868]  req@ffff88082e25ac00 x1457097791263548/t0(0) o104->MGS@192.168.1.42@o2ib:15/16 lens 296/224 e 0 to 1 dl 1389597869 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:24:28 mds1 kernel: Lustre: 4442:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 13 02:24:28 mds1 kernel: LustreError: 4442:0:(ldlm_lockd.c:662:ldlm_handle_ast_error()) ### client (nid 192.168.1.42@o2ib) returned 0 from blocking AST ns: MGS lock: ffff88101f9c1100/0x23b4c8bbe785561d lrc: 4/0,0 mode: CR/CR res: [0x636d65:0x2:0x0].0 rrc: 3 type: PLN flags: 0x60000000000020 nid: 192.168.1.42@o2ib remote: 0xdcb871587be8a3ee expref: 10 pid: 4262 timeout: 4295967960 lvb_type: 0
Jan 13 02:24:43 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 2:30, or until 1 client reconnects
Jan 13 02:24:43 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 02:24:43 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 13 02:24:49 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389597883/real 0]  req@ffff880feefd1400 x1457097791264172/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389597889 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:24:49 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 02:25:08 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 0@lo when existing nid 192.168.1.42@o2ib is already connected
Jan 13 02:25:33 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389597933/real 1389597933]  req@ffff880feeebf000 x1457097791266420/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389597944 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:25:33 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 02:25:58 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-MDT0001_UUID seen on new nid 0@lo when existing nid 192.168.1.42@o2ib is already connected
Jan 13 02:25:58 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 0@lo when existing nid 192.168.1.42@o2ib is already connected
Jan 13 02:25:58 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 02:25:58 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 02:26:39 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389597983/real 1389597985]  req@ffff880feeff0000 x1457097791268384/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389597999 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:26:39 mds1 kernel: Lustre: 3343:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 02:26:40 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 13 02:26:40 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 13 02:26:42 mds1 kernel: Lustre: emc-MDT0000: already connected client emc-MDT0001-mdtlov_UUID (at 192.168.1.42@o2ib) with handle 0xdcb871587be88eaf. Rejecting client with the same UUID trying to reconnect with handle 0xb28a83a2bd1faa3d
Jan 13 02:26:42 mds1 kernel: Lustre: emc-MDT0000: already connected client emc-MDT0000-lwp-MDT0001_UUID (at 192.168.1.42@o2ib) with handle 0xdcb871587be89179. Rejecting client with the same UUID trying to reconnect with handle 0xb28a83a2bd1fad07
Jan 13 02:26:42 mds1 kernel: format at ldlm_lib.c:998:target_handle_connect doesn't end in newline
Jan 13 02:26:48 mds1 kernel: Lustre: emc-MDT0000: Rejecting reconnect from the known client emc-MDT0000-lwp-MDT0001_UUID (at 192.168.1.42@o2ib) because it is indicating it is a new client
Jan 13 02:26:48 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation obd_ping failed with -107.
Jan 13 02:26:48 mds1 kernel: LustreError: Skipped 19 previous similar messages
Jan 13 02:26:48 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 02:27:13 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 13 02:27:28 mds1 kernel: Lustre: MGS: haven't heard from client 36ee3639-ed5d-e188-20c0-411cbc0efea2 (at 192.168.1.42@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff8810332ec000, cur 1389598048 expire 1389597898 last 1389597800
Jan 13 02:30:38 mds1 init: tty (/dev/tty1) main process (3815) killed by TERM signal
Jan 13 02:30:38 mds1 init: tty (/dev/tty2) main process (3817) killed by TERM signal
Jan 13 02:30:38 mds1 init: tty (/dev/tty3) main process (3819) killed by TERM signal
Jan 13 02:30:38 mds1 init: tty (/dev/tty4) main process (3821) killed by TERM signal
Jan 13 02:30:38 mds1 init: tty (/dev/tty5) main process (3823) killed by TERM signal
Jan 13 02:30:38 mds1 init: tty (/dev/tty6) main process (3825) killed by TERM signal
Jan 13 02:30:38 mds1 abrtd: Got signal 15, exiting
Jan 13 02:30:42 mds1 acpid: exiting
Jan 13 02:30:42 mds1 multipathd: mdt2: event checker exit
Jan 13 02:30:42 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 02:30:42 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 02:30:42 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 13 02:30:42 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 13 02:30:42 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 02:30:42 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 02:30:43 mds1 multipathd: mdt1: stop event checker thread (140371148838656)
Jan 13 02:30:43 mds1 multipathd: mgt: stop event checker thread (140371142301440)
Jan 13 02:30:43 mds1 multipathd: --------shut down-------
Jan 13 02:30:43 mds1 init: DisconnecJan 13 02:33:39 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 02:33:39 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3126" x-info="http://www.rsyslog.com"] start
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 02:33:39 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 02:33:39 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 02:33:39 mds1 kernel: KERNEL supported cpus:
Jan 13 02:33:39 mds1 kernel:  Intel GenuineIntel
Jan 13 02:33:39 mds1 kernel:  AMD AuthenticAMD
Jan 13 02:33:39 mds1 kernel:  Centaur CentaurHauls
Jan 13 02:33:39 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 02:33:39 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 02:33:39 mds1 kernel: DMI 2.6 present.
Jan 13 02:33:39 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 02:33:39 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 02:33:39 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 02:33:39 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 02:33:39 mds1 kernel: Using GB pages for direct mapping
Jan 13 02:33:39 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 02:33:39 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 02:33:39 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 02:33:39 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 02:33:39 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 02:33:39 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 02:33:39 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 02:33:39 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:33:39 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:33:39 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:33:39 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 02:33:39 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 02:33:39 mds1 kernel: Setting APIC routing to flat.
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 02:33:39 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 02:33:39 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 02:33:39 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 02:33:39 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 02:33:39 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 02:33:39 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 02:33:39 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 02:33:39 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 02:33:39 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 02:33:39 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 02:33:39 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 02:33:39 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 02:33:39 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 02:33:39 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 02:33:39 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 02:33:39 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 02:33:39 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 02:33:39 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 02:33:39 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 02:33:39 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 02:33:39 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 02:33:39 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 02:33:39 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 02:33:39 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 02:33:39 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 02:33:39 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 02:33:39 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 02:33:39 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 02:33:39 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 02:33:39 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 02:33:39 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 02:33:39 mds1 kernel: Zone PFN ranges:
Jan 13 02:33:39 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 02:33:39 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 02:33:39 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 02:33:39 mds1 kernel: Movable zone start PFN for each node
Jan 13 02:33:39 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 02:33:39 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 02:33:39 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 02:33:39 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 02:33:39 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 02:33:39 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 02:33:39 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 02:33:39 mds1 kernel: Setting APIC routing to flat.
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 02:33:39 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 02:33:39 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 02:33:39 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 02:33:39 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 02:33:39 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 02:33:39 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 02:33:39 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 02:33:39 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 02:33:39 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 02:33:39 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 02:33:39 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 02:33:39 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 02:33:39 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 02:33:39 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 02:33:39 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 02:33:39 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 02:33:39 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 02:33:39 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 02:33:39 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 02:33:39 mds1 kernel: Policy zone: Normal
Jan 13 02:33:39 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 02:33:39 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 02:33:39 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 02:33:39 mds1 kernel: Checking aperture...
Jan 13 02:33:39 mds1 kernel: No AGP bridge found
Jan 13 02:33:39 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 02:33:39 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 02:33:39 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 02:33:39 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 02:33:39 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 02:33:39 mds1 kernel: Hierarchical RCU implementation.
Jan 13 02:33:39 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 02:33:39 mds1 kernel: Extended CMOS year: 2000
Jan 13 02:33:39 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 02:33:39 mds1 kernel: console [tty0] enabled
Jan 13 02:33:39 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 02:33:39 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 02:33:39 mds1 kernel: Fast TSC calibration using PIT
Jan 13 02:33:39 mds1 kernel: Detected 1995.255 MHz processor.
Jan 13 02:33:39 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.51 BogoMIPS (lpj=1995255)
Jan 13 02:33:39 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 02:33:39 mds1 kernel: Security Framework initialized
Jan 13 02:33:39 mds1 kernel: SELinux:  Initializing.
Jan 13 02:33:39 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 02:33:39 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 02:33:39 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys ns
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys memory
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys devices
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 02:33:39 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 02:33:39 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 02:33:39 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 02:33:39 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 02:33:39 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 02:33:39 mds1 kernel: using mwait in idle threads.
Jan 13 02:33:39 mds1 kernel: ACPI: Core revision 20090903
Jan 13 02:33:39 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 02:33:39 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 02:33:39 mds1 kernel: dmar: Host address width 46
Jan 13 02:33:39 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 02:33:39 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 02:33:39 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 02:33:39 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 02:33:39 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 02:33:39 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 02:33:39 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 02:33:39 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 02:33:39 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 02:33:39 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 02:33:39 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 02:33:39 mds1 kernel: Enabling x2apic
Jan 13 02:33:39 mds1 kernel: Enabled x2apic
Jan 13 02:33:39 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 02:33:39 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 02:33:39 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 02:33:39 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 02:33:39 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 02:33:39 mds1 kernel: ... version:                3
Jan 13 02:33:39 mds1 kernel: ... bit width:              48
Jan 13 02:33:39 mds1 kernel: ... generic registers:      8
Jan 13 02:33:39 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 02:33:39 mds1 kernel: ... max period:             000000007fffffff
Jan 13 02:33:39 mds1 kernel: ... fixed-purpose events:   3
Jan 13 02:33:39 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 02:33:39 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 02:33:39 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 02:33:39 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 02:33:39 mds1 kernel: Brought up 12 CPUs
Jan 13 02:33:39 mds1 kernel: Total of 12 processors activated (47882.13 BogoMIPS).
Jan 13 02:33:39 mds1 kernel: devtmpfs: initialized
Jan 13 02:33:39 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 02:33:39 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 02:33:39 mds1 kernel: regulator: core version 0.5
Jan 13 02:33:39 mds1 kernel: NET: Registered protocol family 16
Jan 13 02:33:39 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 02:33:39 mds1 kernel: ACPI: bus type pci registered
Jan 13 02:33:39 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 02:33:39 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 02:33:39 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 02:33:39 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 02:33:39 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 02:33:39 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 02:33:39 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 02:33:39 mds1 kernel: ACPI: Interpreter enabled
Jan 13 02:33:39 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 02:33:39 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 02:33:39 mds1 kernel: ACPI: No dock devices found.
Jan 13 02:33:39 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 02:33:39 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 02:33:39 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 02:33:39 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 02:33:39 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 02:33:39 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:33:39 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 02:33:39 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 02:33:39 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 02:33:39 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 02:33:39 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 02:33:39 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:33:39 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 02:33:39 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 02:33:39 mds1 kernel: vgaarb: loaded
Jan 13 02:33:39 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 02:33:39 mds1 kernel: SCSI subsystem initialized
Jan 13 02:33:39 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 02:33:39 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 02:33:39 mds1 kernel: usbcore: registered new device driver usb
Jan 13 02:33:39 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 02:33:39 mds1 kernel: NetLabel: Initializing
Jan 13 02:33:39 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 02:33:39 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 02:33:39 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 02:33:39 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 02:33:39 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 02:33:39 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 02:33:39 mds1 kernel: Switching to clocksource hpet
Jan 13 02:33:39 mds1 kernel: pnp: PnP ACPI init
Jan 13 02:33:39 mds1 kernel: ACPI: bus type pnp registered
Jan 13 02:33:39 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 02:33:39 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 02:33:39 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 02:33:39 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:33:39 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:33:39 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 02:33:39 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:33:39 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 02:33:39 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:33:39 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:33:39 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 02:33:39 mds1 kernel: NET: Registered protocol family 2
Jan 13 02:33:39 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 02:33:39 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 02:33:39 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 02:33:39 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 02:33:39 mds1 kernel: TCP reno registered
Jan 13 02:33:39 mds1 kernel: NET: Registered protocol family 1
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 02:33:39 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 02:33:39 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 02:33:39 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 02:33:39 mds1 kernel: dmar: Unsupported device scope
Jan 13 02:33:39 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 02:33:39 mds1 kernel: type=2000 audit(1389598397.053:1): initialized
Jan 13 02:33:39 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 02:33:39 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 02:33:39 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 02:33:39 mds1 kernel: msgmni has been set to 32768
Jan 13 02:33:39 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 02:33:39 mds1 kernel: ksign: Installing public key data
Jan 13 02:33:39 mds1 kernel: Loading keyring
Jan 13 02:33:39 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 02:33:39 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 02:33:39 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 02:33:39 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 02:33:39 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 02:33:39 mds1 kernel: io scheduler noop registered
Jan 13 02:33:39 mds1 kernel: io scheduler anticipatory registered
Jan 13 02:33:39 mds1 kernel: io scheduler deadline registered (default)
Jan 13 02:33:39 mds1 kernel: io scheduler cfq registered
Jan 13 02:33:39 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 02:33:39 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 02:33:39 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 02:33:39 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 02:33:39 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 02:33:39 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 02:33:39 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 02:33:39 mds1 kernel: ipmi message handler version 39.2
Jan 13 02:33:39 mds1 kernel: IPMI System Interface driver.
Jan 13 02:33:39 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 02:33:39 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 02:33:39 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 02:33:39 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 02:33:39 mds1 kernel: IPMI kcs interface initialized
Jan 13 02:33:39 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 02:33:39 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 02:33:39 mds1 kernel: ERST: Could not register with persistent store
Jan 13 02:33:39 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 02:33:39 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 02:33:39 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 02:33:39 mds1 kernel: Linux agpgart interface v0.103
Jan 13 02:33:39 mds1 kernel: crash memory driver: version 1.1
Jan 13 02:33:39 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 02:33:39 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 02:33:39 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 02:33:39 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 02:33:39 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 02:33:39 mds1 kernel: brd: module loaded
Jan 13 02:33:39 mds1 kernel: loop: module loaded
Jan 13 02:33:39 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 02:33:39 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 02:33:39 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 02:33:39 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 02:33:39 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 02:33:39 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 02:33:39 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 02:33:39 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 02:33:39 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 02:33:39 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 02:33:39 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 02:33:39 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 02:33:39 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 02:33:39 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 02:33:39 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 02:33:39 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 02:33:39 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 02:33:39 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 02:33:39 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 02:33:39 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 02:33:39 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 02:33:39 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 02:33:39 mds1 kernel: Switching to clocksource tsc
Jan 13 02:33:39 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 02:33:39 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 02:33:39 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 02:33:39 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 02:33:39 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 02:33:39 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 02:33:39 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 02:33:39 mds1 kernel: cpuidle: using governor ladder
Jan 13 02:33:39 mds1 kernel: cpuidle: using governor menu
Jan 13 02:33:39 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 02:33:39 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 02:33:39 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 02:33:39 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 02:33:39 mds1 kernel: TCP cubic registered
Jan 13 02:33:39 mds1 kernel: Initializing XFRM netlink socket
Jan 13 02:33:39 mds1 kernel: NET: Registered protocol family 17
Jan 13 02:33:39 mds1 kernel: registered taskstats version 1
Jan 13 02:33:39 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 07:33:19 UTC (1389598399)
Jan 13 02:33:39 mds1 kernel: Initalizing network drop monitor service
Jan 13 02:33:39 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 02:33:39 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 02:33:39 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 02:33:39 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 02:33:39 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 02:33:39 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 02:33:39 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 02:33:39 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 02:33:39 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 02:33:39 mds1 kernel: udev: starting version 147
Jan 13 02:33:39 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 02:33:39 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 02:33:39 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 02:33:39 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 02:33:39 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 02:33:39 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 02:33:39 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 02:33:39 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 02:33:39 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 02:33:39 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 02:33:39 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 02:33:39 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 02:33:39 mds1 kernel: megasas: FW now in Ready state
Jan 13 02:33:39 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 02:33:39 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 02:33:39 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 02:33:39 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 02:33:39 mds1 kernel: megasas:IOC Init cmd success
Jan 13 02:33:39 mds1 kernel: megasas: INIT adapter done
Jan 13 02:33:39 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 02:33:39 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 02:33:39 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 02:33:39 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 02:33:39 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 02:33:39 mds1 kernel: scsi1 : qla2xxx
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 02:33:39 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 02:33:39 mds1 kernel: scsi2 : qla2xxx
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 02:33:39 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 02:33:39 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 02:33:39 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 02:33:39 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 02:33:39 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 02:33:39 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 02:33:39 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 02:33:39 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 02:33:39 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dc6, mb[3]=e100, cmd=6a ****.
Jan 13 02:33:39 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2de6, mb[3]=a100, cmd=6a ****.
Jan 13 02:33:39 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 02:33:39 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 02:33:39 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 02:33:39 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 02:33:39 mds1 kernel: scsi3 : ahci
Jan 13 02:33:39 mds1 kernel: scsi4 : ahci
Jan 13 02:33:39 mds1 kernel: scsi5 : ahci
Jan 13 02:33:39 mds1 kernel: scsi6 : ahci
Jan 13 02:33:39 mds1 kernel: scsi7 : ahci
Jan 13 02:33:39 mds1 kernel: scsi8 : ahci
Jan 13 02:33:39 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 02:33:39 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 02:33:39 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 02:33:39 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 02:33:39 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 02:33:39 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 02:33:39 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 02:33:39 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 02:33:39 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 02:33:39 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 02:33:39 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 02:33:39 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sda:
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 02:33:39 mds1 kernel: sdc:
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 02:33:39 mds1 kernel: sdd:
Jan 13 02:33:39 mds1 kernel: sdb:
Jan 13 02:33:39 mds1 kernel: sde:
Jan 13 02:33:39 mds1 kernel: sdf: unknown partition table
Jan 13 02:33:39 mds1 kernel: unknown partition table
Jan 13 02:33:39 mds1 kernel: unknown partition table
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: unknown partition table
Jan 13 02:33:39 mds1 kernel: unknown partition table
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: sdg:
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: unknown partition table
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: sda1 sda2 sda3
Jan 13 02:33:39 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 02:33:39 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 02:33:39 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 02:33:39 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 02:33:39 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 02:33:39 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 02:33:39 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 02:33:39 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 02:33:39 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 02:33:39 mds1 kernel: type=1404 audit(1389598402.868:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 02:33:39 mds1 kernel: dracut: 
Jan 13 02:33:39 mds1 kernel: dracut: Switching root
Jan 13 02:33:39 mds1 kernel: udev: starting version 147
Jan 13 02:33:39 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 02:33:39 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 02:33:39 mds1 kernel: PTP clock support registered
Jan 13 02:33:39 mds1 kernel: dca service started, version 1.12.1
Jan 13 02:33:39 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 02:33:39 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 02:33:39 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 02:33:39 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 02:33:39 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 02:33:39 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 02:33:39 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 02:33:39 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 02:33:39 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 02:33:39 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 02:33:39 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 02:33:39 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 02:33:39 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 02:33:39 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 02:33:39 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 02:33:39 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 02:33:39 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 02:33:39 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 02:33:39 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 02:33:39 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 02:33:39 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 02:33:39 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 02:33:39 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 02:33:39 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 02:33:39 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 02:33:39 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:33:39 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 02:33:39 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:33:39 mds1 kernel: emc: device handler registered
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 02:33:39 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 02:33:39 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 02:33:39 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 02:33:39 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 02:33:39 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 02:33:39 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 02:33:39 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 02:33:39 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102888a000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 02:33:39 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102888a000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 02:33:39 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 02:33:39 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 02:33:39 mds1 kernel: NET: Registered protocol family 10
Jan 13 02:33:39 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 02:33:39 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 02:33:39 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 02:33:39 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 02:33:39 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 02:33:39 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:33:39 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 02:33:39 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 02:33:39 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 02:33:39 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 02:33:39 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 02:33:39 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 02:33:39 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 02:33:39 mds1 rpc.statd[3221]: Version 1.2.3 starting
Jan 13 02:33:39 mds1 sm-notify[3222]: Version 1.2.3 starting
Jan 13 02:33:39 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 02:33:39 mds1 kernel: RPC: Registered udp transport module.
Jan 13 02:33:39 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 02:33:39 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 02:33:39 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 02:33:39 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 02:33:39 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 02:33:39 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 02:33:43 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 02:33:43 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 02:33:48 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 02:33:48 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 02:33:48 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 02:33:48 mds1 kernel: LNet: Accept secure, port 988
Jan 13 02:33:48 mds1 cib[3384]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:48 mds1 attrd[3387]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:48 mds1 stonith-ng[3385]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:48 mds1 pengine[3388]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:48 mds1 lrmd[3386]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:48 mds1 crmd[3389]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 02:33:49 mds1 logger[3398]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 02:33:50 mds1 kdump: kexec: loaded kdump kernel
Jan 13 02:33:50 mds1 kdump: started up
Jan 13 02:33:51 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 02:33:51 mds1 kernel: Slow work thread pool: Starting up
Jan 13 02:33:51 mds1 kernel: Slow work thread pool: Ready
Jan 13 02:33:51 mds1 kernel: FS-Cache: Loaded
Jan 13 02:33:51 mds1 kernel: Registering the id_resolver key type
Jan 13 02:33:51 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 02:33:51 mds1 acpid: starting up
Jan 13 02:33:51 mds1 acpid: 1 rule loaded
Jan 13 02:33:51 mds1 acpid: waiting for events: event logging is off
Jan 13 02:33:51 mds1 acpid: client connected from 3615[68:68]
Jan 13 02:33:51 mds1 acpid: 1 client rule loaded
Jan 13 02:33:51 mds1 kernel: ipmi device interface
Jan 13 02:33:53 mds1 automount[3654]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 02:33:53 mds1 abrtd: Init complete, entering main loop
Jan 13 02:33:54 mds1 cimserver[3805]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 02:33:59 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 02:34:01 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 02:34:14 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 02:34:15 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.46@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:34:15 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.46@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:34:15 mds1 kernel: LustreError: Skipped 13 previous similar messages
Jan 13 02:34:18 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 02:34:19 mds1 kernel: LustreError: 4094:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 13 02:34:19 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 02:34:19 mds1 kernel: LustreError: 4280:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 02:34:19 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:34:24 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389598459/real 0]  req@ffff880ff1a0bc00 x1457099561238608/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389598464 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 02:34:24 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff14ccc00 x1457099561238700/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:26 mds1 kernel: LustreError: 4148:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff1bb7c00 x1457099561238704/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:30 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff14ccc00 x1457099561238708/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:30 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 02:34:32 mds1 kernel: LustreError: 4148:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff1bb7c00 x1457099561238712/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:36 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff14ccc00 x1457099561238716/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:36 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 02:34:42 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff1bb7800 x1457099561238724/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:42 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 02:34:53 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff1bb7800 x1457099561238732/t0(0) o253->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 02:34:53 mds1 kernel: LustreError: 4094:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 02:35:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:35:01 mds1 kernel: LustreError: Skipped 25 previous similar messages
Jan 13 02:35:01 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:35:15 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 02:35:15 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 02:35:26 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:35:51 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:35:51 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:36:16 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:36:41 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:36:41 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:37:06 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:37:31 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:37:56 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:37:56 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 02:38:21 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:39:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:39:11 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:39:11 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 02:40:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:41:41 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:41:41 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 02:41:41 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:41:41 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 02:44:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:44:11 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 02:46:16 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 02:46:16 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 13 02:49:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:49:11 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 02:55:01 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 02:55:01 mds1 kernel: LustreError: Skipped 20 previous similar messages
Jan 13 02:58:21 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 02:58:21 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 13 03:05:26 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 03:05:26 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 03:09:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 03:09:11 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 03:15:51 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 03:15:51 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 03:20:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 03:20:01 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 03:26:16 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 03:26:16 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 03:30:51 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 03:30:51 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 03:36:41 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 03:36:41 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 03:41:41 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 03:41:41 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 03:47:06 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 03:47:06 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 03:52:31 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 03:52:31 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 03:57:31 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 03:57:31 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 04:03:21 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:03:21 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 04:07:56 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:07:56 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 04:14:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:14:11 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 04:18:21 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:18:21 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 04:25:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:25:01 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 04:28:46 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:28:46 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 04:35:51 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:35:51 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 04:37:52 mds1 init: tty (/dev/tty1) main process (3830) killed by TERM signal
Jan 13 04:37:52 mds1 init: tty (/dev/tty2) main process (3832) killed by TERM signal
Jan 13 04:37:52 mds1 init: tty (/dev/tty3) main process (3834) killed by TERM signal
Jan 13 04:37:52 mds1 init: tty (/Jan 13 04:40:53 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 04:40:53 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3126" x-info="http://www.rsyslog.com"] start
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 04:40:53 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 04:40:53 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 04:40:53 mds1 kernel: KERNEL supported cpus:
Jan 13 04:40:53 mds1 kernel:  Intel GenuineIntel
Jan 13 04:40:53 mds1 kernel:  AMD AuthenticAMD
Jan 13 04:40:53 mds1 kernel:  Centaur CentaurHauls
Jan 13 04:40:53 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 04:40:53 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 04:40:53 mds1 kernel: DMI 2.6 present.
Jan 13 04:40:53 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 04:40:53 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 04:40:53 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 04:40:53 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 04:40:53 mds1 kernel: Using GB pages for direct mapping
Jan 13 04:40:53 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 04:40:53 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 04:40:53 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 04:40:53 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 04:40:53 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 04:40:53 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 04:40:53 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 04:40:53 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 04:40:53 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 04:40:53 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 04:40:53 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 04:40:53 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 04:40:53 mds1 kernel: Setting APIC routing to flat.
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 04:40:53 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 04:40:53 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 04:40:53 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 04:40:53 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 04:40:53 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 04:40:53 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 04:40:53 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 04:40:53 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 04:40:53 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 04:40:53 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 04:40:53 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 04:40:53 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 04:40:53 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 04:40:53 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 04:40:53 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 04:40:53 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 04:40:53 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 04:40:53 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 04:40:53 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 04:40:53 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 04:40:53 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 04:40:53 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 04:40:53 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 04:40:53 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 04:40:53 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 04:40:53 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 04:40:53 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 04:40:53 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 04:40:53 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 04:40:53 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 04:40:53 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 04:40:53 mds1 kernel: Zone PFN ranges:
Jan 13 04:40:53 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 04:40:53 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 04:40:53 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 04:40:53 mds1 kernel: Movable zone start PFN for each node
Jan 13 04:40:53 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 04:40:53 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 04:40:53 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 04:40:53 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 04:40:53 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 04:40:53 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 04:40:53 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 04:40:53 mds1 kernel: Setting APIC routing to flat.
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 04:40:53 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 04:40:53 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 04:40:53 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 04:40:53 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 04:40:53 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 04:40:53 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 04:40:53 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 04:40:53 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 04:40:53 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 04:40:53 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 04:40:53 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 04:40:53 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 04:40:53 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 04:40:53 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 04:40:53 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 04:40:53 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 04:40:53 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 04:40:53 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 04:40:53 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 04:40:53 mds1 kernel: Policy zone: Normal
Jan 13 04:40:53 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 04:40:53 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 04:40:53 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 04:40:53 mds1 kernel: Checking aperture...
Jan 13 04:40:53 mds1 kernel: No AGP bridge found
Jan 13 04:40:53 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 04:40:53 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 04:40:53 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 04:40:53 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 04:40:53 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 04:40:53 mds1 kernel: Hierarchical RCU implementation.
Jan 13 04:40:53 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 04:40:53 mds1 kernel: Extended CMOS year: 2000
Jan 13 04:40:53 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 04:40:53 mds1 kernel: console [tty0] enabled
Jan 13 04:40:53 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 04:40:53 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 04:40:53 mds1 kernel: Fast TSC calibration using PIT
Jan 13 04:40:53 mds1 kernel: Detected 1995.332 MHz processor.
Jan 13 04:40:53 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.66 BogoMIPS (lpj=1995332)
Jan 13 04:40:53 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 04:40:53 mds1 kernel: Security Framework initialized
Jan 13 04:40:53 mds1 kernel: SELinux:  Initializing.
Jan 13 04:40:53 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 04:40:53 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 04:40:53 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys ns
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys memory
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys devices
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 04:40:53 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 04:40:53 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 04:40:53 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 04:40:53 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 04:40:53 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 04:40:53 mds1 kernel: using mwait in idle threads.
Jan 13 04:40:53 mds1 kernel: ACPI: Core revision 20090903
Jan 13 04:40:53 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 04:40:53 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 04:40:53 mds1 kernel: dmar: Host address width 46
Jan 13 04:40:53 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 04:40:53 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 04:40:53 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 04:40:53 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 04:40:53 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 04:40:53 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 04:40:53 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 04:40:53 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 04:40:53 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 04:40:53 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 04:40:53 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 04:40:53 mds1 kernel: Enabling x2apic
Jan 13 04:40:53 mds1 kernel: Enabled x2apic
Jan 13 04:40:53 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 04:40:53 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 04:40:53 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 04:40:53 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 04:40:53 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 04:40:53 mds1 kernel: ... version:                3
Jan 13 04:40:53 mds1 kernel: ... bit width:              48
Jan 13 04:40:53 mds1 kernel: ... generic registers:      8
Jan 13 04:40:53 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 04:40:53 mds1 kernel: ... max period:             000000007fffffff
Jan 13 04:40:53 mds1 kernel: ... fixed-purpose events:   3
Jan 13 04:40:53 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 04:40:53 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 04:40:53 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 04:40:53 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 04:40:53 mds1 kernel: Brought up 12 CPUs
Jan 13 04:40:53 mds1 kernel: Total of 12 processors activated (47883.12 BogoMIPS).
Jan 13 04:40:53 mds1 kernel: devtmpfs: initialized
Jan 13 04:40:53 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 04:40:53 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 04:40:53 mds1 kernel: regulator: core version 0.5
Jan 13 04:40:53 mds1 kernel: NET: Registered protocol family 16
Jan 13 04:40:53 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 04:40:53 mds1 kernel: ACPI: bus type pci registered
Jan 13 04:40:53 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 04:40:53 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 04:40:53 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 04:40:53 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 04:40:53 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 04:40:53 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 04:40:53 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 04:40:53 mds1 kernel: ACPI: Interpreter enabled
Jan 13 04:40:53 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 04:40:53 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 04:40:53 mds1 kernel: ACPI: No dock devices found.
Jan 13 04:40:53 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 04:40:53 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 04:40:53 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 04:40:53 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 04:40:53 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 04:40:53 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 04:40:53 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 04:40:53 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 04:40:53 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 04:40:53 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 04:40:53 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 04:40:53 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 04:40:53 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 04:40:53 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 04:40:53 mds1 kernel: vgaarb: loaded
Jan 13 04:40:53 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 04:40:53 mds1 kernel: SCSI subsystem initialized
Jan 13 04:40:53 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 04:40:53 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 04:40:53 mds1 kernel: usbcore: registered new device driver usb
Jan 13 04:40:53 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 04:40:53 mds1 kernel: NetLabel: Initializing
Jan 13 04:40:53 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 04:40:53 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 04:40:53 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 04:40:53 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 04:40:53 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 04:40:53 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 04:40:53 mds1 kernel: Switching to clocksource hpet
Jan 13 04:40:53 mds1 kernel: pnp: PnP ACPI init
Jan 13 04:40:53 mds1 kernel: ACPI: bus type pnp registered
Jan 13 04:40:53 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 04:40:53 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 04:40:53 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 04:40:53 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 04:40:53 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 04:40:53 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 04:40:53 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 04:40:53 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 04:40:53 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 04:40:53 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 04:40:53 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 04:40:53 mds1 kernel: NET: Registered protocol family 2
Jan 13 04:40:53 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 04:40:53 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 04:40:53 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 04:40:53 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 04:40:53 mds1 kernel: TCP reno registered
Jan 13 04:40:53 mds1 kernel: NET: Registered protocol family 1
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 04:40:53 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 04:40:53 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 04:40:53 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 04:40:53 mds1 kernel: dmar: Unsupported device scope
Jan 13 04:40:53 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 04:40:53 mds1 kernel: type=2000 audit(1389606031.055:1): initialized
Jan 13 04:40:53 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 04:40:53 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 04:40:53 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 04:40:53 mds1 kernel: msgmni has been set to 32768
Jan 13 04:40:53 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 04:40:53 mds1 kernel: ksign: Installing public key data
Jan 13 04:40:53 mds1 kernel: Loading keyring
Jan 13 04:40:53 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 04:40:53 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 04:40:53 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 04:40:53 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 04:40:53 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 04:40:53 mds1 kernel: io scheduler noop registered
Jan 13 04:40:53 mds1 kernel: io scheduler anticipatory registered
Jan 13 04:40:53 mds1 kernel: io scheduler deadline registered (default)
Jan 13 04:40:53 mds1 kernel: io scheduler cfq registered
Jan 13 04:40:53 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 04:40:53 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 04:40:53 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 04:40:53 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 04:40:53 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 04:40:53 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 04:40:53 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 04:40:53 mds1 kernel: ipmi message handler version 39.2
Jan 13 04:40:53 mds1 kernel: IPMI System Interface driver.
Jan 13 04:40:53 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 04:40:53 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 04:40:53 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 04:40:53 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 04:40:53 mds1 kernel: IPMI kcs interface initialized
Jan 13 04:40:53 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 04:40:53 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 04:40:53 mds1 kernel: ERST: Could not register with persistent store
Jan 13 04:40:53 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 04:40:53 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 04:40:53 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 04:40:53 mds1 kernel: Linux agpgart interface v0.103
Jan 13 04:40:53 mds1 kernel: crash memory driver: version 1.1
Jan 13 04:40:53 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 04:40:53 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 04:40:53 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 04:40:53 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 04:40:53 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 04:40:53 mds1 kernel: brd: module loaded
Jan 13 04:40:53 mds1 kernel: loop: module loaded
Jan 13 04:40:53 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 04:40:53 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 04:40:53 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 04:40:53 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 04:40:53 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 04:40:53 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 04:40:53 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 04:40:53 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 04:40:53 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 04:40:53 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 04:40:53 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 04:40:53 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 04:40:53 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 04:40:53 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 04:40:53 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 04:40:53 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 04:40:53 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 04:40:53 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 04:40:53 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 04:40:53 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 04:40:53 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 04:40:53 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 04:40:53 mds1 kernel: Refined TSC clocksource calibration: 1995.191 MHz.
Jan 13 04:40:53 mds1 kernel: Switching to clocksource tsc
Jan 13 04:40:53 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 04:40:53 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 04:40:53 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 04:40:53 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 04:40:53 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 04:40:53 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 04:40:53 mds1 kernel: cpuidle: using governor ladder
Jan 13 04:40:53 mds1 kernel: cpuidle: using governor menu
Jan 13 04:40:53 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 04:40:53 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 04:40:53 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 04:40:53 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 04:40:53 mds1 kernel: TCP cubic registered
Jan 13 04:40:53 mds1 kernel: Initializing XFRM netlink socket
Jan 13 04:40:53 mds1 kernel: NET: Registered protocol family 17
Jan 13 04:40:53 mds1 kernel: registered taskstats version 1
Jan 13 04:40:53 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 09:40:33 UTC (1389606033)
Jan 13 04:40:53 mds1 kernel: Initalizing network drop monitor service
Jan 13 04:40:53 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 04:40:53 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 04:40:53 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 04:40:53 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 04:40:53 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 04:40:53 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 04:40:53 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 04:40:53 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 04:40:53 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 04:40:53 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 04:40:53 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 04:40:53 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 04:40:53 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 04:40:53 mds1 kernel: udev: starting version 147
Jan 13 04:40:53 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 04:40:53 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 04:40:53 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 04:40:53 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 04:40:53 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 04:40:53 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 04:40:53 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 04:40:53 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 04:40:53 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 04:40:53 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 04:40:53 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 04:40:53 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 04:40:53 mds1 kernel: megasas: FW now in Ready state
Jan 13 04:40:53 mds1 kernel: megasas:IOC Init cmd success
Jan 13 04:40:53 mds1 kernel: megasas: INIT adapter done
Jan 13 04:40:53 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 04:40:53 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 04:40:53 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 04:40:53 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 04:40:53 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 04:40:53 mds1 kernel: scsi1 : qla2xxx
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 04:40:53 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 04:40:53 mds1 kernel: scsi2 : qla2xxx
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 04:40:53 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 04:40:53 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 04:40:53 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 04:40:53 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 04:40:53 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 04:40:53 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 04:40:53 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 04:40:53 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 04:40:53 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dd1, mb[3]=e100, cmd=6a ****.
Jan 13 04:40:53 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dee, mb[3]=e100, cmd=6a ****.
Jan 13 04:40:53 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 04:40:53 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 04:40:53 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 04:40:53 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 04:40:53 mds1 kernel: scsi3 : ahci
Jan 13 04:40:53 mds1 kernel: scsi4 : ahci
Jan 13 04:40:53 mds1 kernel: scsi5 : ahci
Jan 13 04:40:53 mds1 kernel: scsi6 : ahci
Jan 13 04:40:53 mds1 kernel: scsi7 : ahci
Jan 13 04:40:53 mds1 kernel: scsi8 : ahci
Jan 13 04:40:53 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 04:40:53 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 04:40:53 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 04:40:53 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 04:40:53 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 04:40:53 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 04:40:53 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 04:40:53 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 04:40:53 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 04:40:53 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 04:40:53 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 04:40:53 mds1 kernel: sdc:
Jan 13 04:40:53 mds1 kernel: sda:
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 04:40:53 mds1 kernel: sdb:
Jan 13 04:40:53 mds1 kernel: sdd:
Jan 13 04:40:53 mds1 kernel: sde:
Jan 13 04:40:53 mds1 kernel: sdf: unknown partition table
Jan 13 04:40:53 mds1 kernel: unknown partition table
Jan 13 04:40:53 mds1 kernel: unknown partition table
Jan 13 04:40:53 mds1 kernel: unknown partition table
Jan 13 04:40:53 mds1 kernel: unknown partition table
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sdg: unknown partition table
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sda1 sda2 sda3
Jan 13 04:40:53 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 04:40:53 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 04:40:53 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 04:40:53 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 04:40:53 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 04:40:53 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 04:40:53 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 04:40:53 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 04:40:53 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 04:40:53 mds1 kernel: type=1404 audit(1389606036.657:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 04:40:53 mds1 kernel: dracut: 
Jan 13 04:40:53 mds1 kernel: dracut: Switching root
Jan 13 04:40:53 mds1 kernel: udev: starting version 147
Jan 13 04:40:53 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 04:40:53 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 04:40:53 mds1 kernel: PTP clock support registered
Jan 13 04:40:53 mds1 kernel: dca service started, version 1.12.1
Jan 13 04:40:53 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 04:40:53 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 04:40:53 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 04:40:53 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 04:40:53 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 04:40:53 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 04:40:53 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 04:40:53 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 04:40:53 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 04:40:53 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 04:40:53 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 04:40:53 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 04:40:53 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 04:40:53 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 04:40:53 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 04:40:53 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 04:40:53 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 04:40:53 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 04:40:53 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 04:40:53 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 04:40:53 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 04:40:53 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 04:40:53 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 04:40:53 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 04:40:53 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 04:40:53 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 04:40:53 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 04:40:53 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 04:40:53 mds1 kernel: emc: device handler registered
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 04:40:53 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 04:40:53 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 04:40:53 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 04:40:53 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 04:40:53 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 04:40:53 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 04:40:53 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 04:40:53 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10295e4000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 04:40:53 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10295e4000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 04:40:53 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 04:40:53 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 04:40:53 mds1 kernel: NET: Registered protocol family 10
Jan 13 04:40:53 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 04:40:53 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 04:40:53 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 04:40:53 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 04:40:53 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 04:40:53 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 04:40:53 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 04:40:53 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 04:40:53 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 04:40:53 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 04:40:53 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 04:40:53 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 04:40:53 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 04:40:53 mds1 rpc.statd[3221]: Version 1.2.3 starting
Jan 13 04:40:53 mds1 sm-notify[3222]: Version 1.2.3 starting
Jan 13 04:40:53 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 04:40:53 mds1 kernel: RPC: Registered udp transport module.
Jan 13 04:40:53 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 04:40:53 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 04:40:53 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 04:40:53 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 04:40:53 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 04:40:53 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 04:40:58 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 04:40:58 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 04:41:02 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 04:41:02 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 04:41:02 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 04:41:02 mds1 kernel: LNet: Accept secure, port 988
Jan 13 04:41:03 mds1 cib[3384]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 attrd[3387]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 stonith-ng[3385]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 pengine[3388]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 lrmd[3386]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 crmd[3389]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 04:41:03 mds1 logger[3398]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 04:41:05 mds1 kdump: kexec: loaded kdump kernel
Jan 13 04:41:05 mds1 kdump: started up
Jan 13 04:41:05 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 04:41:05 mds1 kernel: Slow work thread pool: Starting up
Jan 13 04:41:05 mds1 kernel: Slow work thread pool: Ready
Jan 13 04:41:05 mds1 kernel: FS-Cache: Loaded
Jan 13 04:41:05 mds1 kernel: Registering the id_resolver key type
Jan 13 04:41:05 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 04:41:05 mds1 acpid: starting up
Jan 13 04:41:05 mds1 acpid: 1 rule loaded
Jan 13 04:41:05 mds1 acpid: waiting for events: event logging is off
Jan 13 04:41:06 mds1 acpid: client connected from 3615[68:68]
Jan 13 04:41:06 mds1 acpid: 1 client rule loaded
Jan 13 04:41:06 mds1 kernel: ipmi device interface
Jan 13 04:41:07 mds1 automount[3655]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 04:41:08 mds1 abrtd: Init complete, entering main loop
Jan 13 04:41:08 mds1 cimserver[3806]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 04:41:14 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 04:41:16 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 04:41:24 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 04:41:29 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 04:41:30 mds1 kernel: LustreError: 4093:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 13 04:41:30 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 04:41:30 mds1 kernel: LustreError: 4278:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 04:41:30 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:41:35 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389606090/real 0]  req@ffff880812d17000 x1457107566067792/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389606095 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 04:41:36 mds1 kernel: LustreError: 4093:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880815523800 x1457107566067884/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:41:36 mds1 kernel: LustreError: 4147:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881005904c00 x1457107566067888/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:41:42 mds1 kernel: LustreError: 4093:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880815523800 x1457107566067892/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:41:42 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 04:41:48 mds1 kernel: LustreError: 4093:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880815523800 x1457107566067900/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:41:48 mds1 kernel: LustreError: 4093:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 04:41:48 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 04:41:54 mds1 kernel: LustreError: 4093:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880815523800 x1457107566067908/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:41:54 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 04:42:01 mds1 kernel: LustreError: 4147:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881031f89c00 x1457107566067920/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:42:01 mds1 kernel: LustreError: 4147:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 04:42:01 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 04:42:13 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:42:13 mds1 kernel: LustreError: 4147:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff881031f89c00 x1457107566067928/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 04:42:13 mds1 kernel: LustreError: 4147:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Jan 13 04:42:13 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:42:15 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 04:42:15 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 04:42:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:43:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:43:03 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:43:28 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:43:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:43:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:44:18 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:44:43 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:45:08 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:45:08 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 04:45:33 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:46:23 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:46:23 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:46:23 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 04:47:13 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:48:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:48:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:48:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 04:48:53 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 04:50:33 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:50:33 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 04:53:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:53:03 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 04:53:28 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 04:53:28 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 13 04:58:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 04:58:03 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 05:02:13 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 05:02:13 mds1 kernel: LustreError: Skipped 20 previous similar messages
Jan 13 05:07:13 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 05:07:13 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 13 05:12:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 05:12:38 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 05:18:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 05:18:03 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 05:23:03 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 05:23:03 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 05:28:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 05:28:53 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 05:33:28 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 05:33:28 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 05:39:43 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 05:39:43 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 05:43:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 05:43:53 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 05:50:33 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 05:50:33 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 05:54:18 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 05:54:18 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:01:23 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:01:23 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:04:43 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 06:04:43 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:12:13 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:12:13 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:15:08 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 06:15:08 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:23:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:23:03 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:25:33 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 06:25:33 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:33:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:33:53 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:35:58 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 06:35:58 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:44:43 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:44:43 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:46:23 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 06:46:23 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 06:55:33 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 06:55:33 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 06:56:48 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 06:56:48 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 07:06:23 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 07:06:23 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 07:07:13 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 07:07:13 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 07:17:13 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 07:17:13 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 07:17:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 07:17:38 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 07:28:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 07:28:03 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 07:28:03 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 07:28:03 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 07:38:28 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 07:38:28 mds1 kernel: LustreError: Skipped 24 previous similar messages
Jan 13 07:38:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 07:38:53 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 13 07:43:43 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 07:43:43 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 07:43:43 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 13 07:43:45 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection to emc-OST0010 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 07:43:45 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 13 07:44:24 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617058/real 0]  req@ffff88102161a400 x1457107566282880/t0(0) o8->emc-OST0013-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617064 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:44:24 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 07:44:28 mds1 init: tty (/dev/tty1) main process (3831) killed by TERM signal
Jan 13 07:44:28 mds1 init: tty (/dev/tty2) main process (3833) killed by TERM signal
Jan 13 07:44:28 mds1 init: tty (/dev/tty3) main process (3835) killed by TERM signal
Jan 13 07:44:28 mds1 init: tty (/dev/tty4) main process (3837) killed by TERM signal
Jan 13 07:44:28 mds1 init: tty (/dev/tty5) main process (3839) killed by TERM signal
Jan 13 07:44:28 mds1 init: tty (/dev/tty6) main process (3841) killed by TERM signal
Jan 13 07:44:29 mds1 abrtd: Got signal 15, exiting
Jan 13 07:44:32 mds1 acpid: exiting
Jan 13 07:44:32 mds1 multipathd: mdt2: event checker exit
Jan 13 07:44:32 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 07:44:32 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 07:44:32 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 13 07:44:32 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 13 07:44:32 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 07:44:32 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 07:44:34 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 07:44:35 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 07:44:40 mds1 kernel: Lustre: 25618:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389617074/real 1389617074]  req@ffff88082e6d9c00 x1457107566282920/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389617080 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:44:40 mds1 kernel: Lustre: 25618:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 07:44:41 mds1 kernel: Lustre: server umount MGS complete
Jan 13 07:44:43 mds1 multipathd: mgt: stop event checker thread (139817775642368)
Jan 13 07:44:43 mds1 multipathd: mdt1: stop event checker thread (139817764898560)
Jan 13 07:44:43 mds1 multipathd: --------shut down-------
Jan 13 07:44:43 mds1 init: Disconnected from system bus
Jan 13 07:44:43 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 13 07:44:43 mds1 auditd[3101]: The audit daemon is exiting.
Jan 13 07:44:43 mds1 kernel: type=1305 audit(1389617083.521:7288): audit_pid=0 old=3101 auid=4294967295 ses=4294967295 res=1
Jan 13 07:44:43 mds1 kernel: type=1305 audit(1389617083.622:7289): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Jan 13 07:44:43 mds1 kernel: Kernel logging (proc) stopped.
Jan 13 07:44:43 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3126" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 13 07:50:26 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 07:50:26 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3097" x-info="http://www.rsyslog.com"] start
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 07:50:26 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 07:50:26 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 07:50:26 mds1 kernel: KERNEL supported cpus:
Jan 13 07:50:26 mds1 kernel:  Intel GenuineIntel
Jan 13 07:50:26 mds1 kernel:  AMD AuthenticAMD
Jan 13 07:50:26 mds1 kernel:  Centaur CentaurHauls
Jan 13 07:50:26 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 07:50:26 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 07:50:26 mds1 kernel: DMI 2.6 present.
Jan 13 07:50:26 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 07:50:26 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 07:50:26 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 07:50:26 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 07:50:26 mds1 kernel: Using GB pages for direct mapping
Jan 13 07:50:26 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 07:50:26 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 07:50:26 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 07:50:26 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 07:50:26 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 07:50:26 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 07:50:26 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 07:50:26 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 07:50:26 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 07:50:26 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 07:50:26 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 07:50:26 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 07:50:26 mds1 kernel: Setting APIC routing to flat.
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 07:50:26 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 07:50:26 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 07:50:26 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 07:50:26 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 07:50:26 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 07:50:26 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 07:50:26 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 07:50:26 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 07:50:26 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 07:50:26 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 07:50:26 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 07:50:26 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 07:50:26 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 07:50:26 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 07:50:26 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 07:50:26 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 07:50:26 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 07:50:26 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 07:50:26 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 07:50:26 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 07:50:26 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 07:50:26 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 07:50:26 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 07:50:26 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 07:50:26 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 07:50:26 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 07:50:26 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 07:50:26 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 07:50:26 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 07:50:26 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 07:50:26 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 07:50:26 mds1 kernel: Zone PFN ranges:
Jan 13 07:50:26 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 07:50:26 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 07:50:26 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 07:50:26 mds1 kernel: Movable zone start PFN for each node
Jan 13 07:50:26 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 07:50:26 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 07:50:26 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 07:50:26 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 07:50:26 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 07:50:26 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 07:50:26 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 07:50:26 mds1 kernel: Setting APIC routing to flat.
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 07:50:26 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 07:50:26 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 07:50:26 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 07:50:26 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 07:50:26 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 07:50:26 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 07:50:26 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 07:50:26 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 07:50:26 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 07:50:26 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 07:50:26 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 07:50:26 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 07:50:26 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 07:50:26 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 07:50:26 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 07:50:26 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 07:50:26 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 07:50:26 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 07:50:26 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 07:50:26 mds1 kernel: Policy zone: Normal
Jan 13 07:50:26 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 07:50:26 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 07:50:26 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 07:50:26 mds1 kernel: Checking aperture...
Jan 13 07:50:26 mds1 kernel: No AGP bridge found
Jan 13 07:50:26 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 07:50:26 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 07:50:26 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 07:50:26 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 07:50:26 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 07:50:26 mds1 kernel: Hierarchical RCU implementation.
Jan 13 07:50:26 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 07:50:26 mds1 kernel: Extended CMOS year: 2000
Jan 13 07:50:26 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 07:50:26 mds1 kernel: console [tty0] enabled
Jan 13 07:50:26 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 07:50:26 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 07:50:26 mds1 kernel: Fast TSC calibration using PIT
Jan 13 07:50:26 mds1 kernel: Detected 1995.210 MHz processor.
Jan 13 07:50:26 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.42 BogoMIPS (lpj=1995210)
Jan 13 07:50:26 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 07:50:26 mds1 kernel: Security Framework initialized
Jan 13 07:50:26 mds1 kernel: SELinux:  Initializing.
Jan 13 07:50:26 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 07:50:26 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 07:50:26 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys ns
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys memory
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys devices
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 07:50:26 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 07:50:26 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 07:50:26 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 07:50:26 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 07:50:26 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 07:50:26 mds1 kernel: using mwait in idle threads.
Jan 13 07:50:26 mds1 kernel: ACPI: Core revision 20090903
Jan 13 07:50:26 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 07:50:26 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 07:50:26 mds1 kernel: dmar: Host address width 46
Jan 13 07:50:26 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 07:50:26 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 07:50:26 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 07:50:26 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 07:50:26 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 07:50:26 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 07:50:26 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 07:50:26 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 07:50:26 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 07:50:26 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 07:50:26 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 07:50:26 mds1 kernel: Enabling x2apic
Jan 13 07:50:26 mds1 kernel: Enabled x2apic
Jan 13 07:50:26 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 07:50:26 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 07:50:26 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 07:50:26 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 07:50:26 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 07:50:26 mds1 kernel: ... version:                3
Jan 13 07:50:26 mds1 kernel: ... bit width:              48
Jan 13 07:50:26 mds1 kernel: ... generic registers:      8
Jan 13 07:50:26 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 07:50:26 mds1 kernel: ... max period:             000000007fffffff
Jan 13 07:50:26 mds1 kernel: ... fixed-purpose events:   3
Jan 13 07:50:26 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 07:50:26 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 07:50:26 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 07:50:26 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 07:50:26 mds1 kernel: Brought up 12 CPUs
Jan 13 07:50:26 mds1 kernel: Total of 12 processors activated (47881.59 BogoMIPS).
Jan 13 07:50:26 mds1 kernel: devtmpfs: initialized
Jan 13 07:50:26 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 07:50:26 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 07:50:26 mds1 kernel: regulator: core version 0.5
Jan 13 07:50:26 mds1 kernel: NET: Registered protocol family 16
Jan 13 07:50:26 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 07:50:26 mds1 kernel: ACPI: bus type pci registered
Jan 13 07:50:26 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 07:50:26 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 07:50:26 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 07:50:26 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 07:50:26 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 07:50:26 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 07:50:26 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 07:50:26 mds1 kernel: ACPI: Interpreter enabled
Jan 13 07:50:26 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 07:50:26 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 07:50:26 mds1 kernel: ACPI: No dock devices found.
Jan 13 07:50:26 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 07:50:26 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 07:50:26 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 07:50:26 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 07:50:26 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 07:50:26 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 07:50:26 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 07:50:26 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 07:50:26 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 07:50:26 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 07:50:26 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 07:50:26 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 07:50:26 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 07:50:26 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 07:50:26 mds1 kernel: vgaarb: loaded
Jan 13 07:50:26 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 07:50:26 mds1 kernel: SCSI subsystem initialized
Jan 13 07:50:26 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 07:50:26 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 07:50:26 mds1 kernel: usbcore: registered new device driver usb
Jan 13 07:50:26 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 07:50:26 mds1 kernel: NetLabel: Initializing
Jan 13 07:50:26 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 07:50:26 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 07:50:26 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 07:50:26 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 07:50:26 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 07:50:26 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 07:50:26 mds1 kernel: Switching to clocksource hpet
Jan 13 07:50:26 mds1 kernel: pnp: PnP ACPI init
Jan 13 07:50:26 mds1 kernel: ACPI: bus type pnp registered
Jan 13 07:50:26 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 07:50:26 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 07:50:26 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 07:50:26 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 07:50:26 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 07:50:26 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 07:50:26 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 07:50:26 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 07:50:26 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 07:50:26 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 07:50:26 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 07:50:26 mds1 kernel: NET: Registered protocol family 2
Jan 13 07:50:26 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 07:50:26 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 07:50:26 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 07:50:26 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 07:50:26 mds1 kernel: TCP reno registered
Jan 13 07:50:26 mds1 kernel: NET: Registered protocol family 1
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 07:50:26 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 07:50:26 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 07:50:26 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 07:50:26 mds1 kernel: dmar: Unsupported device scope
Jan 13 07:50:26 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 07:50:26 mds1 kernel: type=2000 audit(1389617404.055:1): initialized
Jan 13 07:50:26 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 07:50:26 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 07:50:26 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 07:50:26 mds1 kernel: msgmni has been set to 32768
Jan 13 07:50:26 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 07:50:26 mds1 kernel: ksign: Installing public key data
Jan 13 07:50:26 mds1 kernel: Loading keyring
Jan 13 07:50:26 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 07:50:26 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 07:50:26 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 07:50:26 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 07:50:26 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 07:50:26 mds1 kernel: io scheduler noop registered
Jan 13 07:50:26 mds1 kernel: io scheduler anticipatory registered
Jan 13 07:50:26 mds1 kernel: io scheduler deadline registered (default)
Jan 13 07:50:26 mds1 kernel: io scheduler cfq registered
Jan 13 07:50:26 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 07:50:26 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 07:50:26 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 07:50:26 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 07:50:26 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 07:50:26 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 07:50:26 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 07:50:26 mds1 kernel: ipmi message handler version 39.2
Jan 13 07:50:26 mds1 kernel: IPMI System Interface driver.
Jan 13 07:50:26 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 07:50:26 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 07:50:26 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 07:50:26 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 07:50:26 mds1 kernel: IPMI kcs interface initialized
Jan 13 07:50:26 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 07:50:26 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 07:50:26 mds1 kernel: ERST: Could not register with persistent store
Jan 13 07:50:26 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 07:50:26 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 07:50:26 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 07:50:26 mds1 kernel: Linux agpgart interface v0.103
Jan 13 07:50:26 mds1 kernel: crash memory driver: version 1.1
Jan 13 07:50:26 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 07:50:26 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 07:50:26 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 07:50:26 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 07:50:26 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 07:50:26 mds1 kernel: brd: module loaded
Jan 13 07:50:26 mds1 kernel: loop: module loaded
Jan 13 07:50:26 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 07:50:26 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 07:50:26 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 07:50:26 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 07:50:26 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 07:50:26 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 07:50:26 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 07:50:26 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 07:50:26 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 07:50:26 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 07:50:26 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 07:50:26 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 07:50:26 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 07:50:26 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 07:50:26 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 07:50:26 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 07:50:26 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 07:50:26 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 07:50:26 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 07:50:26 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 07:50:26 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 07:50:26 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 07:50:26 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 07:50:26 mds1 kernel: Switching to clocksource tsc
Jan 13 07:50:26 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 07:50:26 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 07:50:26 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 07:50:26 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 07:50:26 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 07:50:26 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 07:50:26 mds1 kernel: cpuidle: using governor ladder
Jan 13 07:50:26 mds1 kernel: cpuidle: using governor menu
Jan 13 07:50:26 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 07:50:26 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 07:50:26 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 07:50:26 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 07:50:26 mds1 kernel: TCP cubic registered
Jan 13 07:50:26 mds1 kernel: Initializing XFRM netlink socket
Jan 13 07:50:26 mds1 kernel: NET: Registered protocol family 17
Jan 13 07:50:26 mds1 kernel: registered taskstats version 1
Jan 13 07:50:26 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 12:50:06 UTC (1389617406)
Jan 13 07:50:26 mds1 kernel: Initalizing network drop monitor service
Jan 13 07:50:26 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 07:50:26 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 07:50:26 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 07:50:26 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 07:50:26 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 07:50:26 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 07:50:26 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 07:50:26 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 07:50:26 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 07:50:26 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 07:50:26 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 07:50:26 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 07:50:26 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 07:50:26 mds1 kernel: udev: starting version 147
Jan 13 07:50:26 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 07:50:26 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 07:50:26 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 07:50:26 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 07:50:26 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 07:50:26 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 07:50:26 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 07:50:26 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 07:50:26 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 07:50:26 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 07:50:26 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 07:50:26 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 07:50:26 mds1 kernel: megasas: FW now in Ready state
Jan 13 07:50:26 mds1 kernel: megasas:IOC Init cmd success
Jan 13 07:50:26 mds1 kernel: megasas: INIT adapter done
Jan 13 07:50:26 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 07:50:26 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 07:50:26 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 07:50:26 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 07:50:26 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 07:50:26 mds1 kernel: scsi1 : qla2xxx
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 07:50:26 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 07:50:26 mds1 kernel: scsi2 : qla2xxx
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 07:50:26 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 07:50:26 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 07:50:26 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 07:50:26 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 07:50:26 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 07:50:26 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 07:50:26 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 07:50:26 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 07:50:26 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=303e, mb[3]=2100, cmd=6a ****.
Jan 13 07:50:26 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2fae, mb[3]=7100, cmd=6a ****.
Jan 13 07:50:26 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 07:50:26 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 07:50:26 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 07:50:26 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 07:50:26 mds1 kernel: scsi3 : ahci
Jan 13 07:50:26 mds1 kernel: scsi4 : ahci
Jan 13 07:50:26 mds1 kernel: scsi5 : ahci
Jan 13 07:50:26 mds1 kernel: scsi6 : ahci
Jan 13 07:50:26 mds1 kernel: scsi7 : ahci
Jan 13 07:50:26 mds1 kernel: scsi8 : ahci
Jan 13 07:50:26 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 07:50:26 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 07:50:26 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 07:50:26 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 07:50:26 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 07:50:26 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 07:50:26 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 07:50:26 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 07:50:26 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 07:50:26 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 07:50:26 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 07:50:26 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 07:50:26 mds1 kernel: sda:
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 07:50:26 mds1 kernel: sdc:
Jan 13 07:50:26 mds1 kernel: sdd:
Jan 13 07:50:26 mds1 kernel: sdb:
Jan 13 07:50:26 mds1 kernel: sde: unknown partition table
Jan 13 07:50:26 mds1 kernel: unknown partition table
Jan 13 07:50:26 mds1 kernel: sdg:
Jan 13 07:50:26 mds1 kernel: sdf: unknown partition table
Jan 13 07:50:26 mds1 kernel: unknown partition table
Jan 13 07:50:26 mds1 kernel: unknown partition table
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: unknown partition table
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sda1 sda2 sda3
Jan 13 07:50:26 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 07:50:26 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 07:50:26 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 07:50:26 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 07:50:26 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 07:50:26 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 07:50:26 mds1 kernel: type=1404 audit(1389617409.343:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 07:50:26 mds1 kernel: dracut: 
Jan 13 07:50:26 mds1 kernel: dracut: Switching root
Jan 13 07:50:26 mds1 kernel: readahead: starting
Jan 13 07:50:26 mds1 kernel: udev: starting version 147
Jan 13 07:50:26 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 07:50:26 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 07:50:26 mds1 kernel: PTP clock support registered
Jan 13 07:50:26 mds1 kernel: dca service started, version 1.12.1
Jan 13 07:50:26 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 07:50:26 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 07:50:26 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 07:50:26 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 07:50:26 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 07:50:26 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 07:50:26 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 07:50:26 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 07:50:26 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 07:50:26 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 07:50:26 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 07:50:26 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 07:50:26 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 07:50:26 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 07:50:26 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 07:50:26 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 07:50:26 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 07:50:26 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 07:50:26 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 07:50:26 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 07:50:26 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 07:50:26 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 07:50:26 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 07:50:26 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 07:50:26 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 07:50:26 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 07:50:26 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 07:50:26 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 07:50:26 mds1 kernel: emc: device handler registered
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 07:50:26 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 07:50:26 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 07:50:26 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 07:50:26 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 07:50:26 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 07:50:26 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 07:50:26 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 07:50:26 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028dc6000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 07:50:26 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028dc6000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 07:50:26 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 07:50:26 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 07:50:26 mds1 kernel: NET: Registered protocol family 10
Jan 13 07:50:26 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 07:50:26 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 07:50:26 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 07:50:26 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 07:50:26 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 07:50:26 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 07:50:26 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 07:50:26 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 07:50:26 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 07:50:26 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 07:50:26 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 07:50:26 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 07:50:26 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 07:50:26 mds1 rpc.statd[3192]: Version 1.2.3 starting
Jan 13 07:50:26 mds1 sm-notify[3193]: Version 1.2.3 starting
Jan 13 07:50:26 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 07:50:26 mds1 kernel: RPC: Registered udp transport module.
Jan 13 07:50:26 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 07:50:26 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 07:50:26 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 07:50:26 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 07:50:26 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 07:50:26 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 07:50:30 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 07:50:30 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 07:50:34 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 07:50:34 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 07:50:34 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 07:50:34 mds1 kernel: LNet: Accept secure, port 988
Jan 13 07:50:35 mds1 cib[3355]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:35 mds1 attrd[3358]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:35 mds1 pengine[3359]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:35 mds1 crmd[3360]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:35 mds1 stonith-ng[3356]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:35 mds1 lrmd[3357]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 07:50:36 mds1 logger[3369]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 07:50:37 mds1 kdump: kexec: loaded kdump kernel
Jan 13 07:50:37 mds1 kdump: started up
Jan 13 07:50:39 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 07:50:39 mds1 kernel: Slow work thread pool: Starting up
Jan 13 07:50:39 mds1 kernel: Slow work thread pool: Ready
Jan 13 07:50:39 mds1 kernel: FS-Cache: Loaded
Jan 13 07:50:39 mds1 kernel: Registering the id_resolver key type
Jan 13 07:50:39 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 07:50:39 mds1 acpid: starting up
Jan 13 07:50:39 mds1 acpid: 1 rule loaded
Jan 13 07:50:39 mds1 acpid: waiting for events: event logging is off
Jan 13 07:50:40 mds1 acpid: client connected from 3589[68:68]
Jan 13 07:50:40 mds1 acpid: 1 client rule loaded
Jan 13 07:50:40 mds1 kernel: ipmi device interface
Jan 13 07:50:41 mds1 automount[3628]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 07:50:42 mds1 abrtd: Init complete, entering main loop
Jan 13 07:50:42 mds1 cimserver[3779]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 07:50:46 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 07:50:48 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 07:51:21 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 07:51:21 mds1 kernel: 
Jan 13 07:51:21 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 07:51:21 mds1 kernel: 
Jan 13 07:51:21 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 07:51:21 mds1 kernel: 
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 07:52:03 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 07:52:04 mds1 kernel: LustreError: 4155:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -108 (not fatal)
Jan 13 07:52:04 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 07:52:04 mds1 kernel: LustreError: 4378:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 07:52:04 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 07:52:04 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 07:52:04 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 07:52:09 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617524/real 0]  req@ffff8807efef1c00 x1457119490474100/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617529 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:52:14 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617529/real 0]  req@ffff8807efef1800 x1457119490474128/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389617534 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:52:14 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
Jan 13 07:52:14 mds1 kernel: LustreError: 4157:0:(ldlm_resource.c:809:ldlm_resource_complain()) MGC192.168.1.40@o2ib: namespace resource [0x636d65:0x0:0x0].0 (ffff880827fa2e80) refcount nonzero (1) after lock cleanup; forcing cleanup.
Jan 13 07:52:14 mds1 kernel: LustreError: 4157:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x636d65:0x0:0x0].0 (ffff880827fa2e80) refcount = 2
Jan 13 07:52:14 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 07:52:14 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 13 07:52:14 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 07:52:19 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617534/real 0]  req@ffff8807ed181c00 x1457119490474352/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617539 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:52:39 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 07:52:39 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 07:52:39 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 13 07:52:44 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617559/real 0]  req@ffff8807ecd97000 x1457119490474704/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617564 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:52:44 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 07:53:04 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 07:53:04 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 07:53:04 mds1 kernel: Lustre: emc-MDT0000-osp-MDT0001: Connection restored to emc-MDT0000 (at 0@lo)
Jan 13 07:53:14 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617584/real 0]  req@ffff880827edc400 x1457119490474920/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617594 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:53:14 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 48 previous similar messages
Jan 13 07:53:39 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617609/real 0]  req@ffff880827c6c400 x1457119490475132/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617619 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:53:39 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 07:54:09 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389617634/real 0]  req@ffff8807ecd7ac00 x1457119490475344/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389617649 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:54:09 mds1 kernel: Lustre: 3328:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 07:54:24 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 13 07:54:24 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_disconnect failed with -19.
Jan 13 07:54:24 mds1 kernel: LustreError: 5025:0:(osp_dev.c:338:osp_disconnect()) emc-MDT0000-osp-MDT0001: can't disconnect: rc = -19
Jan 13 07:54:24 mds1 kernel: LustreError: 5025:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031: rc = -19
Jan 13 07:54:24 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 07:54:34 mds1 kernel: Lustre: 5024:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389617664/real 1389617664]  req@ffff8807ece2dc00 x1457119490475592/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389617674 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 07:54:34 mds1 kernel: Lustre: 5024:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 07:54:34 mds1 kernel: Lustre: server umount MGS complete
Jan 13 07:54:34 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 07:56:18 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 07:56:20 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 07:56:20 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 07:56:20 mds1 kernel: LustreError: 5434:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 11, won't add
Jan 13 07:56:20 mds1 kernel: LustreError: 5434:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 07:56:20 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 07:56:20 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 07:56:20 mds1 kernel: LustreError: 5434:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 07:56:20 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 07:56:20 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 07:56:20 mds1 kernel: LustreError: 5367:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 07:56:20 mds1 kernel: LustreError: 5367:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 07:56:20 mds1 kernel: LustreError: 5367:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 07:56:20 mds1 kernel: LustreError: 5367:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 07:56:20 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 07:56:20 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 07:56:20 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 07:56:20 mds1 kernel: LustreError: 5367:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 08:50:02 mds1 init: tty (/dev/tty1) main process (3804) killed by TERM signal
Jan 13 08:50:02 mds1 init: tty (/dev/tty2) main process (3806) killed by TERM signal
Jan 13 08:50:02 mds1 init: tty (/dev/tty3) main process (3808) killed by TERM signal
Jan 13 08:50:02 mds1 init: tty (/dev/tty4) main process (3810) killed by TERM signal
Jan 13 08:50:02 mds1 init: tty (/dev/tty5) main process (3812) killed by TERM signal
Jan 13 08:50:02 mds1 init: tty (/dev/tty6) main process (3814) killed by TERM signal
Jan 13 08:50:02 mds1 abrtd: Got signal 15, exiting
Jan 13 08:50:06 mds1 acpid: exiting
Jan 13 08:50:06 mds1 multipathd: mdt1: event checker exit
Jan 13 08:50:06 mds1 multipathd: dm-0: remove map (uevent)
Jan 13 08:50:06 mds1 multipathd: dm-0: devmap not registered, can't remove
Jan 13 08:50:06 mds1 kernel: sd 1:0:0:1: emc: Detached
Jan 13 08:50:06 mds1 kernel: sd 2:0:0:1: emc: Detached
Jan 13 08:50:06 mds1 multipathd: dm-0: remove map (uevent)
Jan 13 08:50:06 mds1 multipathd: dm-0: devmap not registered, can't remove
Jan 13 08:50:06 mds1 multipathd: mdt2: event checker exit
Jan 13 08:50:06 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 08:50:06 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 08:50:06 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 13 08:50:06 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 13 08:50:06 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 08:50:06 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 08:52:59 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 08:52:59 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3139" x-info="http://www.rsyslog.com"] start
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 08:52:59 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 08:52:59 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 08:52:59 mds1 kernel: KERNEL supported cpus:
Jan 13 08:52:59 mds1 kernel:  Intel GenuineIntel
Jan 13 08:52:59 mds1 kernel:  AMD AuthenticAMD
Jan 13 08:52:59 mds1 kernel:  Centaur CentaurHauls
Jan 13 08:52:59 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 08:52:59 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 08:52:59 mds1 kernel: DMI 2.6 present.
Jan 13 08:52:59 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 08:52:59 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 08:52:59 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 08:52:59 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 08:52:59 mds1 kernel: Using GB pages for direct mapping
Jan 13 08:52:59 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 08:52:59 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 08:52:59 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 08:52:59 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 08:52:59 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 08:52:59 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 08:52:59 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 08:52:59 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 08:52:59 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 08:52:59 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 08:52:59 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 08:52:59 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 08:52:59 mds1 kernel: Setting APIC routing to flat.
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 08:52:59 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 08:52:59 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 08:52:59 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 08:52:59 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 08:52:59 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 08:52:59 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 08:52:59 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 08:52:59 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 08:52:59 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 08:52:59 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 08:52:59 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 08:52:59 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 08:52:59 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 08:52:59 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 08:52:59 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 08:52:59 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 08:52:59 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 08:52:59 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 08:52:59 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 08:52:59 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 08:52:59 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 08:52:59 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 08:52:59 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 08:52:59 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 08:52:59 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 08:52:59 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 08:52:59 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 08:52:59 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 08:52:59 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 08:52:59 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 08:52:59 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 08:52:59 mds1 kernel: Zone PFN ranges:
Jan 13 08:52:59 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 08:52:59 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 08:52:59 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 08:52:59 mds1 kernel: Movable zone start PFN for each node
Jan 13 08:52:59 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 08:52:59 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 08:52:59 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 08:52:59 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 08:52:59 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 08:52:59 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 08:52:59 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 08:52:59 mds1 kernel: Setting APIC routing to flat.
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 08:52:59 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 08:52:59 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 08:52:59 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 08:52:59 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 08:52:59 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 08:52:59 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 08:52:59 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 08:52:59 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 08:52:59 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 08:52:59 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 08:52:59 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 08:52:59 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 08:52:59 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 08:52:59 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 08:52:59 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 08:52:59 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 08:52:59 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 08:52:59 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 08:52:59 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 08:52:59 mds1 kernel: Policy zone: Normal
Jan 13 08:52:59 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 08:52:59 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 08:52:59 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 08:52:59 mds1 kernel: Checking aperture...
Jan 13 08:52:59 mds1 kernel: No AGP bridge found
Jan 13 08:52:59 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 08:52:59 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 08:52:59 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 08:52:59 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 08:52:59 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 08:52:59 mds1 kernel: Hierarchical RCU implementation.
Jan 13 08:52:59 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 08:52:59 mds1 kernel: Extended CMOS year: 2000
Jan 13 08:52:59 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 08:52:59 mds1 kernel: console [tty0] enabled
Jan 13 08:52:59 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 08:52:59 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 08:52:59 mds1 kernel: Fast TSC calibration using PIT
Jan 13 08:52:59 mds1 kernel: Detected 1995.055 MHz processor.
Jan 13 08:52:59 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.11 BogoMIPS (lpj=1995055)
Jan 13 08:52:59 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 08:52:59 mds1 kernel: Security Framework initialized
Jan 13 08:52:59 mds1 kernel: SELinux:  Initializing.
Jan 13 08:52:59 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 08:52:59 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 08:52:59 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys ns
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys memory
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys devices
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 08:52:59 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 08:52:59 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 08:52:59 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 08:52:59 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 08:52:59 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 08:52:59 mds1 kernel: using mwait in idle threads.
Jan 13 08:52:59 mds1 kernel: ACPI: Core revision 20090903
Jan 13 08:52:59 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 08:52:59 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 08:52:59 mds1 kernel: dmar: Host address width 46
Jan 13 08:52:59 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 08:52:59 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 08:52:59 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 08:52:59 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 08:52:59 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 08:52:59 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 08:52:59 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 08:52:59 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 08:52:59 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 08:52:59 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 08:52:59 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 08:52:59 mds1 kernel: Enabling x2apic
Jan 13 08:52:59 mds1 kernel: Enabled x2apic
Jan 13 08:52:59 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 08:52:59 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 08:52:59 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 08:52:59 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 08:52:59 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 08:52:59 mds1 kernel: ... version:                3
Jan 13 08:52:59 mds1 kernel: ... bit width:              48
Jan 13 08:52:59 mds1 kernel: ... generic registers:      8
Jan 13 08:52:59 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 08:52:59 mds1 kernel: ... max period:             000000007fffffff
Jan 13 08:52:59 mds1 kernel: ... fixed-purpose events:   3
Jan 13 08:52:59 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 08:52:59 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 08:52:59 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 08:52:59 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 08:52:59 mds1 kernel: Brought up 12 CPUs
Jan 13 08:52:59 mds1 kernel: Total of 12 processors activated (47879.73 BogoMIPS).
Jan 13 08:52:59 mds1 kernel: devtmpfs: initialized
Jan 13 08:52:59 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 08:52:59 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 08:52:59 mds1 kernel: regulator: core version 0.5
Jan 13 08:52:59 mds1 kernel: NET: Registered protocol family 16
Jan 13 08:52:59 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 08:52:59 mds1 kernel: ACPI: bus type pci registered
Jan 13 08:52:59 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 08:52:59 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 08:52:59 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 08:52:59 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 08:52:59 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 08:52:59 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 08:52:59 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 08:52:59 mds1 kernel: ACPI: Interpreter enabled
Jan 13 08:52:59 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 08:52:59 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 08:52:59 mds1 kernel: ACPI: No dock devices found.
Jan 13 08:52:59 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 08:52:59 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 08:52:59 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 08:52:59 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 08:52:59 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 08:52:59 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 08:52:59 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 08:52:59 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 08:52:59 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 08:52:59 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 08:52:59 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 08:52:59 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 08:52:59 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 08:52:59 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 08:52:59 mds1 kernel: vgaarb: loaded
Jan 13 08:52:59 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 08:52:59 mds1 kernel: SCSI subsystem initialized
Jan 13 08:52:59 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 08:52:59 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 08:52:59 mds1 kernel: usbcore: registered new device driver usb
Jan 13 08:52:59 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 08:52:59 mds1 kernel: NetLabel: Initializing
Jan 13 08:52:59 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 08:52:59 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 08:52:59 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 08:52:59 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 08:52:59 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 08:52:59 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 08:52:59 mds1 kernel: Switching to clocksource hpet
Jan 13 08:52:59 mds1 kernel: pnp: PnP ACPI init
Jan 13 08:52:59 mds1 kernel: ACPI: bus type pnp registered
Jan 13 08:52:59 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 08:52:59 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 08:52:59 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 08:52:59 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 08:52:59 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 08:52:59 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 08:52:59 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 08:52:59 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 08:52:59 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 08:52:59 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 08:52:59 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 08:52:59 mds1 kernel: NET: Registered protocol family 2
Jan 13 08:52:59 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 08:52:59 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 08:52:59 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 08:52:59 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 08:52:59 mds1 kernel: TCP reno registered
Jan 13 08:52:59 mds1 kernel: NET: Registered protocol family 1
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 08:52:59 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 08:52:59 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 08:52:59 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 08:52:59 mds1 kernel: dmar: Unsupported device scope
Jan 13 08:52:59 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 08:52:59 mds1 kernel: type=2000 audit(1389621157.054:1): initialized
Jan 13 08:52:59 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 08:52:59 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 08:52:59 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 08:52:59 mds1 kernel: msgmni has been set to 32768
Jan 13 08:52:59 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 08:52:59 mds1 kernel: ksign: Installing public key data
Jan 13 08:52:59 mds1 kernel: Loading keyring
Jan 13 08:52:59 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 08:52:59 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 08:52:59 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 08:52:59 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 08:52:59 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 08:52:59 mds1 kernel: io scheduler noop registered
Jan 13 08:52:59 mds1 kernel: io scheduler anticipatory registered
Jan 13 08:52:59 mds1 kernel: io scheduler deadline registered (default)
Jan 13 08:52:59 mds1 kernel: io scheduler cfq registered
Jan 13 08:52:59 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 08:52:59 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 08:52:59 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 08:52:59 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 08:52:59 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 08:52:59 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 08:52:59 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 08:52:59 mds1 kernel: ipmi message handler version 39.2
Jan 13 08:52:59 mds1 kernel: IPMI System Interface driver.
Jan 13 08:52:59 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 08:52:59 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 08:52:59 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 08:52:59 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 08:52:59 mds1 kernel: IPMI kcs interface initialized
Jan 13 08:52:59 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 08:52:59 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 08:52:59 mds1 kernel: ERST: Could not register with persistent store
Jan 13 08:52:59 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 08:52:59 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 08:52:59 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 08:52:59 mds1 kernel: Linux agpgart interface v0.103
Jan 13 08:52:59 mds1 kernel: crash memory driver: version 1.1
Jan 13 08:52:59 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 08:52:59 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 08:52:59 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 08:52:59 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 08:52:59 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 08:52:59 mds1 kernel: brd: module loaded
Jan 13 08:52:59 mds1 kernel: loop: module loaded
Jan 13 08:52:59 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 08:52:59 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 08:52:59 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 08:52:59 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 08:52:59 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 08:52:59 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 08:52:59 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 08:52:59 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 08:52:59 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 08:52:59 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 08:52:59 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 08:52:59 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 08:52:59 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 08:52:59 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 08:52:59 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 08:52:59 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 08:52:59 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 08:52:59 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 08:52:59 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 08:52:59 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 08:52:59 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 08:52:59 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 08:52:59 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 08:52:59 mds1 kernel: Switching to clocksource tsc
Jan 13 08:52:59 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 08:52:59 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 08:52:59 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 08:52:59 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 08:52:59 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 08:52:59 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 08:52:59 mds1 kernel: cpuidle: using governor ladder
Jan 13 08:52:59 mds1 kernel: cpuidle: using governor menu
Jan 13 08:52:59 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 08:52:59 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 08:52:59 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 08:52:59 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 08:52:59 mds1 kernel: TCP cubic registered
Jan 13 08:52:59 mds1 kernel: Initializing XFRM netlink socket
Jan 13 08:52:59 mds1 kernel: NET: Registered protocol family 17
Jan 13 08:52:59 mds1 kernel: registered taskstats version 1
Jan 13 08:52:59 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 13:52:39 UTC (1389621159)
Jan 13 08:52:59 mds1 kernel: Initalizing network drop monitor service
Jan 13 08:52:59 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 08:52:59 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 08:52:59 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 08:52:59 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 08:52:59 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 08:52:59 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 08:52:59 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 08:52:59 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 08:52:59 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 08:52:59 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 08:52:59 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 08:52:59 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 08:52:59 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 08:52:59 mds1 kernel: udev: starting version 147
Jan 13 08:52:59 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 08:52:59 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 08:52:59 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 08:52:59 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 08:52:59 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 08:52:59 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 08:52:59 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 08:52:59 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 08:52:59 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 08:52:59 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 08:52:59 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 08:52:59 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 08:52:59 mds1 kernel: megasas: FW now in Ready state
Jan 13 08:52:59 mds1 kernel: megasas:IOC Init cmd success
Jan 13 08:52:59 mds1 kernel: megasas: INIT adapter done
Jan 13 08:52:59 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 08:52:59 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 08:52:59 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 08:52:59 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 08:52:59 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 08:52:59 mds1 kernel: scsi1 : qla2xxx
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 08:52:59 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 08:52:59 mds1 kernel: scsi2 : qla2xxx
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 08:52:59 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 08:52:59 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 08:52:59 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 08:52:59 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 08:52:59 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 08:52:59 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 08:52:59 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 08:52:59 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 08:52:59 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dd1, mb[3]=8100, cmd=6a ****.
Jan 13 08:52:59 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dee, mb[3]=7100, cmd=6a ****.
Jan 13 08:52:59 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 08:52:59 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 08:52:59 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 08:52:59 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 08:52:59 mds1 kernel: scsi3 : ahci
Jan 13 08:52:59 mds1 kernel: scsi4 : ahci
Jan 13 08:52:59 mds1 kernel: scsi5 : ahci
Jan 13 08:52:59 mds1 kernel: scsi6 : ahci
Jan 13 08:52:59 mds1 kernel: scsi7 : ahci
Jan 13 08:52:59 mds1 kernel: scsi8 : ahci
Jan 13 08:52:59 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 08:52:59 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 08:52:59 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 08:52:59 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 08:52:59 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 08:52:59 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 08:52:59 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 08:52:59 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 08:52:59 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 08:52:59 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 08:52:59 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 08:52:59 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 08:52:59 mds1 kernel: sda:
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sdb:
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 08:52:59 mds1 kernel: sdf:
Jan 13 08:52:59 mds1 kernel: sdd:
Jan 13 08:52:59 mds1 kernel: sdc: unknown partition table
Jan 13 08:52:59 mds1 kernel: unknown partition table
Jan 13 08:52:59 mds1 kernel: unknown partition table
Jan 13 08:52:59 mds1 kernel: unknown partition table
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: sdg:
Jan 13 08:52:59 mds1 kernel: sde:
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: unknown partition table
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: unknown partition table
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: sda1 sda2 sda3
Jan 13 08:52:59 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 08:52:59 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 08:52:59 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 08:52:59 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 08:52:59 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 08:52:59 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 08:52:59 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 08:52:59 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 08:52:59 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 08:52:59 mds1 kernel: type=1404 audit(1389621162.519:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 08:52:59 mds1 kernel: dracut: 
Jan 13 08:52:59 mds1 kernel: dracut: Switching root
Jan 13 08:52:59 mds1 kernel: udev: starting version 147
Jan 13 08:52:59 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 08:52:59 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 08:52:59 mds1 kernel: PTP clock support registered
Jan 13 08:52:59 mds1 kernel: dca service started, version 1.12.1
Jan 13 08:52:59 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 08:52:59 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 08:52:59 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 08:52:59 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 08:52:59 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 08:52:59 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 08:52:59 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 08:52:59 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 08:52:59 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 08:52:59 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 08:52:59 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 08:52:59 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 08:52:59 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 08:52:59 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 08:52:59 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 08:52:59 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 08:52:59 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 08:52:59 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 08:52:59 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 08:52:59 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 08:52:59 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 08:52:59 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 08:52:59 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 08:52:59 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 08:52:59 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 08:52:59 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 08:52:59 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 08:52:59 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 08:52:59 mds1 kernel: emc: device handler registered
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 08:52:59 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 08:52:59 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 08:52:59 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 08:52:59 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 08:52:59 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 08:52:59 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 08:52:59 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 08:52:59 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028e24000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 08:52:59 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028e24000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 08:52:59 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 08:52:59 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 08:52:59 mds1 kernel: NET: Registered protocol family 10
Jan 13 08:52:59 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 08:52:59 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 08:52:59 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 08:52:59 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 08:52:59 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 08:52:59 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 08:52:59 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 08:52:59 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 08:52:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 08:52:59 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 08:52:59 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 08:52:59 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 08:52:59 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 08:52:59 mds1 rpc.statd[3234]: Version 1.2.3 starting
Jan 13 08:52:59 mds1 sm-notify[3235]: Version 1.2.3 starting
Jan 13 08:52:59 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 08:52:59 mds1 kernel: RPC: Registered udp transport module.
Jan 13 08:52:59 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 08:52:59 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 08:52:59 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 08:52:59 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 08:52:59 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 08:52:59 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 08:53:03 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 08:53:03 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 08:53:07 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 08:53:08 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 08:53:08 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 08:53:08 mds1 kernel: LNet: Accept secure, port 988
Jan 13 08:53:08 mds1 attrd[3400]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:08 mds1 cib[3397]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:08 mds1 stonith-ng[3398]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:08 mds1 pengine[3401]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:09 mds1 lrmd[3399]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:09 mds1 crmd[3402]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 08:53:09 mds1 logger[3411]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 08:53:10 mds1 kdump: kexec: loaded kdump kernel
Jan 13 08:53:10 mds1 kdump: started up
Jan 13 08:53:11 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 08:53:11 mds1 kernel: Slow work thread pool: Starting up
Jan 13 08:53:11 mds1 kernel: Slow work thread pool: Ready
Jan 13 08:53:11 mds1 kernel: FS-Cache: Loaded
Jan 13 08:53:11 mds1 kernel: Registering the id_resolver key type
Jan 13 08:53:11 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 08:53:11 mds1 acpid: starting up
Jan 13 08:53:11 mds1 acpid: 1 rule loaded
Jan 13 08:53:11 mds1 acpid: waiting for events: event logging is off
Jan 13 08:53:12 mds1 acpid: client connected from 3627[68:68]
Jan 13 08:53:12 mds1 acpid: 1 client rule loaded
Jan 13 08:53:12 mds1 kernel: ipmi device interface
Jan 13 08:53:13 mds1 automount[3666]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 08:53:14 mds1 abrtd: Init complete, entering main loop
Jan 13 08:53:14 mds1 cimserver[3817]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 08:53:19 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 08:53:21 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 08:53:45 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 08:53:45 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 08:53:45 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 08:53:50 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621225/real 0]  req@ffff880fe4df1c00 x1457123426828296/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389621230 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:53:50 mds1 kernel: LustreError: 4191:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 13 08:53:50 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 08:53:50 mds1 kernel: LustreError: 4385:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 08:53:50 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 08:53:50 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 08:53:50 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 08:53:50 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 08:53:50 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 08:53:50 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 08:53:50 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 08:53:50 mds1 kernel: LustreError: 4454:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 08:53:55 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621230/real 0]  req@ffff880821b40c00 x1457123426828692/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621235 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:54:15 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 08:54:15 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 08:54:15 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 13 08:54:20 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621255/real 0]  req@ffff880fe2760400 x1457123426829044/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621260 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:54:20 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 48 previous similar messages
Jan 13 08:54:40 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 08:54:40 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 08:54:40 mds1 kernel: Lustre: emc-MDT0000-osp-MDT0001: Connection restored to emc-MDT0000 (at 0@lo)
Jan 13 08:54:50 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621280/real 0]  req@ffff880fe273d400 x1457123426829260/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621290 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:54:50 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 49 previous similar messages
Jan 13 08:55:15 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621305/real 0]  req@ffff88082d97f000 x1457123426829472/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621315 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:55:15 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 08:55:45 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621330/real 0]  req@ffff8808214fe400 x1457123426829680/t0(0) o8->emc-OST0016-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621345 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:55:45 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 08:56:10 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621355/real 0]  req@ffff880821722800 x1457123426829896/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621370 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:56:10 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 08:56:40 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389621380/real 0]  req@ffff8808216cf000 x1457123426830108/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389621400 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 08:56:40 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 08:56:47 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 13 08:56:47 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_disconnect failed with -19.
Jan 13 08:56:47 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 08:56:47 mds1 kernel: LustreError: 5052:0:(osp_dev.c:338:osp_disconnect()) emc-MDT0000-osp-MDT0001: can't disconnect: rc = -19
Jan 13 08:56:47 mds1 kernel: LustreError: 5052:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031: rc = -19
Jan 13 08:56:48 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 13 08:56:58 mds1 kernel: Lustre: server umount MGS complete
Jan 13 08:56:58 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 08:58:41 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 08:58:43 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 08:58:43 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 08:58:43 mds1 kernel: LustreError: 5463:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 40, won't add
Jan 13 08:58:43 mds1 kernel: LustreError: 5463:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 08:58:43 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 08:58:43 mds1 kernel: LustreError: 5463:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 08:58:43 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 08:58:43 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 08:58:43 mds1 kernel: LustreError: 5396:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 08:58:43 mds1 kernel: LustreError: 5396:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 08:58:43 mds1 kernel: LustreError: 5396:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 08:58:43 mds1 kernel: LustreError: 5396:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 08:58:43 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 08:58:43 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 08:58:43 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 08:58:43 mds1 kernel: LustreError: 5396:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 09:02:11 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:02:11 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:02:11 mds1 kernel: LustreError: 5926:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 40, won't add
Jan 13 09:02:11 mds1 kernel: LustreError: 5926:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 09:02:11 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 09:02:11 mds1 kernel: LustreError: 5926:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 09:02:11 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 09:02:11 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 09:02:11 mds1 kernel: LustreError: 5856:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 09:02:11 mds1 kernel: LustreError: 5856:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 09:02:11 mds1 kernel: LustreError: 5856:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 09:02:11 mds1 kernel: LustreError: 5856:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 09:02:11 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 09:02:12 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 09:02:12 mds1 kernel: LustreError: 5856:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 09:02:16 mds1 cibadmin[5937]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:02:16 mds1 cibadmin[5938]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:02:40 mds1 cibadmin[5977]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:02:40 mds1 cibadmin[5978]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:02:54 mds1 cibadmin[6027]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:02:54 mds1 cibadmin[6028]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:02:55 mds1 cibadmin[6032]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:02:55 mds1 cibadmin[6033]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:03:16 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:03:16 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:03:16 mds1 kernel: LustreError: 6131:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 40, won't add
Jan 13 09:03:16 mds1 kernel: LustreError: 6131:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 09:03:16 mds1 kernel: LustreError: 6131:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 09:03:16 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 09:03:16 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 09:03:16 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 09:03:16 mds1 kernel: LustreError: 6066:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 09:03:16 mds1 kernel: LustreError: 6066:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 09:03:16 mds1 kernel: LustreError: 6066:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 09:03:16 mds1 kernel: LustreError: 6066:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 09:03:16 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 09:03:17 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 09:03:17 mds1 kernel: LustreError: 6066:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 09:04:12 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:04:12 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:04:12 mds1 kernel: LustreError: 6255:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 40, won't add
Jan 13 09:04:12 mds1 kernel: LustreError: 6255:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 09:04:12 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 09:04:12 mds1 kernel: LustreError: 6255:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 09:04:12 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 09:04:12 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 09:04:12 mds1 kernel: LustreError: 6190:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 09:04:12 mds1 kernel: LustreError: 6190:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 09:04:12 mds1 kernel: LustreError: 6190:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 09:04:12 mds1 kernel: LustreError: 6190:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 09:04:12 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 09:04:12 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 09:04:12 mds1 kernel: LustreError: 6190:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 09:04:38 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:04:38 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:04:38 mds1 kernel: LustreError: 6326:0:(genops.c:320:class_newdev()) Device emc-OST000c-osc-MDT0000 already exists at 40, won't add
Jan 13 09:04:38 mds1 kernel: LustreError: 6326:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000c-osc-MDT0000 of type osp : -17
Jan 13 09:04:38 mds1 kernel: LustreError: 6326:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 13 09:04:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 09:04:38 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000c-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 13 09:04:38 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 13 09:04:38 mds1 kernel: LustreError: 6261:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 13 09:04:38 mds1 kernel: LustreError: 6261:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 13 09:04:38 mds1 kernel: LustreError: 6261:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 13 09:04:38 mds1 kernel: LustreError: 6261:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 13 09:04:38 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 09:04:39 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 09:04:39 mds1 kernel: LustreError: 6261:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 13 09:07:20 mds1 cibadmin[6554]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 13 09:07:50 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:07:50 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:07:50 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 09:07:50 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 09:07:50 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 09:07:50 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.46@o2ib)
Jan 13 09:07:50 mds1 kernel: Lustre: emc-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 13 09:07:54 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 09:07:54 mds1 kernel: LustreError: 6707:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0000: Aborting recovery
Jan 13 09:07:54 mds1 kernel: Lustre: 6705:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 13 09:07:54 mds1 kernel: LustreError: 6668:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 13 09:07:54 mds1 kernel: LustreError: 6670:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0001-osc-MDT0000: can't precreate: rc = -5
Jan 13 09:07:54 mds1 kernel: LustreError: 6670:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0001-osc-MDT0000: cannot precreate objects: rc = -5
Jan 13 09:07:54 mds1 kernel: LustreError: 6672:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0002-osc-MDT0000: can't precreate: rc = -5
Jan 13 09:07:54 mds1 kernel: LustreError: 6672:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0002-osc-MDT0000: cannot precreate objects: rc = -5
Jan 13 09:07:54 mds1 kernel: LustreError: 6666:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST000a-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 13 09:07:55 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 09:08:05 mds1 cibadmin[6732]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:08:05 mds1 cibadmin[6733]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:08:09 mds1 cibadmin[6745]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 09:08:09 mds1 cibadmin[6746]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 09:08:52 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 09:08:52 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 09:08:52 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 09:08:52 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 09:08:52 mds1 kernel: Lustre: emc-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 13 09:09:17 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 09:09:17 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 09:09:17 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 09:09:42 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 13 09:10:16 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 2:30, or until 1 client reconnects
Jan 13 09:10:16 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 09:15:11 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 13 09:15:11 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 09:15:21 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 13 09:29:11 mds1 kernel: Lustre: 8071:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389623344/real 1389623344]  req@ffff880805e29800 x1457123426864132/t0(0) o104->emc-MDT0000@192.168.1.57@o2ib:15/16 lens 296/224 e 0 to 1 dl 1389623351 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 09:29:11 mds1 kernel: Lustre: 8071:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 48 previous similar messages
Jan 13 09:29:11 mds1 kernel: LustreError: 138-a: emc-MDT0000: A client on nid 192.168.1.57@o2ib was evicted due to a lock blocking callback time out: rc -107
Jan 13 09:31:07 mds1 kernel: Lustre: MGS: haven't heard from client f4d7bbbc-d571-2783-802c-e8671d3137de (at 192.168.1.57@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101c9dd000, cur 1389623467 expire 1389623317 last 1389623240
Jan 13 09:31:17 mds1 kernel: Lustre: 6994:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389623470/real 0]  req@ffff8808317ed400 x1457123426867732/t0(0) o104->emc-MDT0000@192.168.1.55@o2ib:15/16 lens 296/224 e 0 to 1 dl 1389623477 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 09:31:17 mds1 kernel: Lustre: 6994:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 09:31:17 mds1 kernel: LustreError: 138-a: emc-MDT0000: A client on nid 192.168.1.55@o2ib was evicted due to a lock blocking callback time out: rc -107
Jan 13 09:31:17 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 09:33:01 mds1 kernel: Lustre: MGS: haven't heard from client 9789c7c8-6881-2f4a-5518-4ab6d3f97b0e (at 192.168.1.55@o2ib) in 231 seconds. I think it's dead, and I am evicting it. exp ffff88101cfed400, cur 1389623581 expire 1389623431 last 1389623350
Jan 13 11:22:12 mds1 kernel: Lustre: MGS: haven't heard from client 02cfa093-cb5f-6039-6315-f740904a7db8 (at 192.168.1.57@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff88081317a800, cur 1389630132 expire 1389629982 last 1389629902
Jan 13 11:22:17 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 9dfd736b-3279-d98d-3531-608d831cbe54 (at 192.168.1.57@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff880813015800, cur 1389630137 expire 1389629987 last 1389629907
Jan 13 11:33:04 mds1 kernel: Lustre: 3375:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630777/real 0]  req@ffff880fe749d800 x1457123427020756/t0(0) o13->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389630784 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 11:33:04 mds1 kernel: Lustre: 3375:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 11:33:04 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 11:33:04 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection to emc-OST0003 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 11:33:04 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 11:33:04 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 11:33:04 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 13 11:33:10 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630784/real 0]  req@ffff880fe88c6000 x1457123427020856/t0(0) o8->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389630790 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:33:10 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 13 11:34:01 mds1 kernel: Lustre: 3379:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630834/real 0]  req@ffff880fe9ee6000 x1457123427021412/t0(0) o400->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1389630841 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:34:01 mds1 kernel: Lustre: 3379:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 13 11:34:01 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 11:34:01 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 11:34:07 mds1 init: tty (/dev/tty1) main process (3842) killed by INT signal
Jan 13 11:34:07 mds1 init: tty (/dev/tty1) main process ended, respawning
Jan 13 11:34:07 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630841/real 0]  req@ffff880fea14e400 x1457123427021560/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389630847 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:34:07 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 11:34:16 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:34:17 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.52@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:34:21 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:34:26 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:34:30 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.53@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:35:02 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630891/real 0]  req@ffff880fea735400 x1457123427022108/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389630902 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:35:06 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:35:06 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 11:35:16 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:35:16 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 11:35:32 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 1b529818-849e-d485-3a63-d83f22bfee9a (at 192.168.1.50@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808216e2400, cur 1389630932 expire 1389630782 last 1389630705
Jan 13 11:35:56 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:35:56 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 13 11:35:57 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630941/real 0]  req@ffff881020b7e800 x1457123427022708/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389630957 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:35:57 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 11:36:46 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:36:46 mds1 kernel: LustreError: Skipped 6 previous similar messages
Jan 13 11:36:52 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389630991/real 0]  req@ffff88101c871c00 x1457123427023308/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389631012 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:36:52 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 11:37:15 mds1 init: tty (/dev/tty2) main process ended, respawning
Jan 13 11:37:21 mds1 kernel: Lustre: MGS: haven't heard from client 78091ffa-2094-1ba6-732e-76884b5ac58b (at 192.168.1.42@o2ib) in 231 seconds. I think it's dead, and I am evicting it. exp ffff881034354400, cur 1389631041 expire 1389630891 last 1389630810
Jan 13 11:37:21 mds1 kernel: Lustre: Skipped 14 previous similar messages
Jan 13 11:38:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:38:01 mds1 kernel: LustreError: Skipped 6 previous similar messages
Jan 13 11:38:17 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389631066/real 0]  req@ffff88101caeb800 x1457123427024200/t0(0) o8->emc-OST0004-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389631097 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:38:17 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jan 13 11:40:27 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389631191/real 0]  req@ffff880ff9588c00 x1457123427025556/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389631227 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:40:27 mds1 kernel: Lustre: 3370:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 13 11:40:31 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:40:31 mds1 kernel: LustreError: Skipped 13 previous similar messages
Jan 13 11:49:54 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 11:49:54 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3149" x-info="http://www.rsyslog.com"] start
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 11:49:54 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 11:49:54 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 11:49:54 mds1 kernel: KERNEL supported cpus:
Jan 13 11:49:54 mds1 kernel:  Intel GenuineIntel
Jan 13 11:49:54 mds1 kernel:  AMD AuthenticAMD
Jan 13 11:49:54 mds1 kernel:  Centaur CentaurHauls
Jan 13 11:49:54 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 11:49:54 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 11:49:54 mds1 kernel: DMI 2.6 present.
Jan 13 11:49:54 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 11:49:54 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 11:49:54 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 11:49:54 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 11:49:54 mds1 kernel: Using GB pages for direct mapping
Jan 13 11:49:54 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 11:49:54 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 11:49:54 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 11:49:54 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 11:49:54 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 11:49:54 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 11:49:54 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 11:49:54 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 11:49:54 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 11:49:54 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 11:49:54 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 11:49:54 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 11:49:54 mds1 kernel: Setting APIC routing to flat.
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 11:49:54 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 11:49:54 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 11:49:54 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 11:49:54 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 11:49:54 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 11:49:54 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 11:49:54 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 11:49:54 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 11:49:54 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 11:49:54 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 11:49:54 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 11:49:54 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 11:49:54 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 11:49:54 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 11:49:54 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 11:49:54 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 11:49:54 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 11:49:54 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 11:49:54 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 11:49:54 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 11:49:54 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 11:49:54 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 11:49:54 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 11:49:54 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 11:49:54 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 11:49:54 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 11:49:54 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 11:49:54 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 11:49:54 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 11:49:54 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 11:49:54 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 11:49:54 mds1 kernel: Zone PFN ranges:
Jan 13 11:49:54 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 11:49:54 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 11:49:54 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 11:49:54 mds1 kernel: Movable zone start PFN for each node
Jan 13 11:49:54 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 11:49:54 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 11:49:54 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 11:49:54 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 11:49:54 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 11:49:54 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 11:49:54 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 11:49:54 mds1 kernel: Setting APIC routing to flat.
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 11:49:54 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 11:49:54 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 11:49:54 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 11:49:54 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 11:49:54 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 11:49:54 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 11:49:54 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 11:49:54 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 11:49:54 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 11:49:54 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 11:49:54 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 11:49:54 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 11:49:54 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 11:49:54 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 11:49:54 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 11:49:54 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 11:49:54 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 11:49:54 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 11:49:54 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 11:49:54 mds1 kernel: Policy zone: Normal
Jan 13 11:49:54 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 11:49:54 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 11:49:54 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 11:49:54 mds1 kernel: Checking aperture...
Jan 13 11:49:54 mds1 kernel: No AGP bridge found
Jan 13 11:49:54 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 11:49:54 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 11:49:54 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 11:49:54 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 11:49:54 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 11:49:54 mds1 kernel: Hierarchical RCU implementation.
Jan 13 11:49:54 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 11:49:54 mds1 kernel: Extended CMOS year: 2000
Jan 13 11:49:54 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 11:49:54 mds1 kernel: console [tty0] enabled
Jan 13 11:49:54 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 11:49:54 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 11:49:54 mds1 kernel: Fast TSC calibration using PIT
Jan 13 11:49:54 mds1 kernel: Detected 1995.054 MHz processor.
Jan 13 11:49:54 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.10 BogoMIPS (lpj=1995054)
Jan 13 11:49:54 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 11:49:54 mds1 kernel: Security Framework initialized
Jan 13 11:49:54 mds1 kernel: SELinux:  Initializing.
Jan 13 11:49:54 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 11:49:54 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 11:49:54 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys ns
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys memory
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys devices
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 11:49:54 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 11:49:54 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 11:49:54 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 11:49:54 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 11:49:54 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 11:49:54 mds1 kernel: using mwait in idle threads.
Jan 13 11:49:54 mds1 kernel: ACPI: Core revision 20090903
Jan 13 11:49:54 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 11:49:54 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 11:49:54 mds1 kernel: dmar: Host address width 46
Jan 13 11:49:54 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 11:49:54 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 11:49:54 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 11:49:54 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 11:49:54 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 11:49:54 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 11:49:54 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 11:49:54 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 11:49:54 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 11:49:54 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 11:49:54 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 11:49:54 mds1 kernel: Enabling x2apic
Jan 13 11:49:54 mds1 kernel: Enabled x2apic
Jan 13 11:49:54 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 11:49:54 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 11:49:54 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 11:49:54 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 11:49:54 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 11:49:54 mds1 kernel: ... version:                3
Jan 13 11:49:54 mds1 kernel: ... bit width:              48
Jan 13 11:49:54 mds1 kernel: ... generic registers:      8
Jan 13 11:49:54 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 11:49:54 mds1 kernel: ... max period:             000000007fffffff
Jan 13 11:49:54 mds1 kernel: ... fixed-purpose events:   3
Jan 13 11:49:54 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 11:49:54 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 11:49:54 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 11:49:54 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 11:49:54 mds1 kernel: Brought up 12 CPUs
Jan 13 11:49:54 mds1 kernel: Total of 12 processors activated (47879.77 BogoMIPS).
Jan 13 11:49:54 mds1 kernel: devtmpfs: initialized
Jan 13 11:49:54 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 11:49:54 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 11:49:54 mds1 kernel: regulator: core version 0.5
Jan 13 11:49:54 mds1 kernel: NET: Registered protocol family 16
Jan 13 11:49:54 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 11:49:54 mds1 kernel: ACPI: bus type pci registered
Jan 13 11:49:54 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 11:49:54 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 11:49:54 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 11:49:54 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 11:49:54 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 11:49:54 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 11:49:54 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 11:49:54 mds1 kernel: ACPI: Interpreter enabled
Jan 13 11:49:54 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 11:49:54 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 11:49:54 mds1 kernel: ACPI: No dock devices found.
Jan 13 11:49:54 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 11:49:54 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 11:49:54 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 11:49:54 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 11:49:54 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 11:49:54 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 11:49:54 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 11:49:54 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 11:49:54 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 11:49:54 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 11:49:54 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 11:49:54 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 11:49:54 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 11:49:54 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 11:49:54 mds1 kernel: vgaarb: loaded
Jan 13 11:49:54 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 11:49:54 mds1 kernel: SCSI subsystem initialized
Jan 13 11:49:54 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 11:49:54 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 11:49:54 mds1 kernel: usbcore: registered new device driver usb
Jan 13 11:49:54 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 11:49:54 mds1 kernel: NetLabel: Initializing
Jan 13 11:49:54 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 11:49:54 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 11:49:54 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 11:49:54 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 11:49:54 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 11:49:54 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 11:49:54 mds1 kernel: Switching to clocksource hpet
Jan 13 11:49:54 mds1 kernel: pnp: PnP ACPI init
Jan 13 11:49:54 mds1 kernel: ACPI: bus type pnp registered
Jan 13 11:49:54 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 11:49:54 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 11:49:54 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 11:49:54 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 11:49:54 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 11:49:54 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 11:49:54 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 11:49:54 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 11:49:54 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 11:49:54 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 11:49:54 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 11:49:54 mds1 kernel: NET: Registered protocol family 2
Jan 13 11:49:54 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 11:49:54 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 11:49:54 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 11:49:54 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 11:49:54 mds1 kernel: TCP reno registered
Jan 13 11:49:54 mds1 kernel: NET: Registered protocol family 1
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 11:49:54 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 11:49:54 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 11:49:54 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 11:49:54 mds1 kernel: dmar: Unsupported device scope
Jan 13 11:49:54 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 11:49:54 mds1 kernel: type=2000 audit(1389631772.054:1): initialized
Jan 13 11:49:54 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 11:49:54 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 11:49:54 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 11:49:54 mds1 kernel: msgmni has been set to 32768
Jan 13 11:49:54 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 11:49:54 mds1 kernel: ksign: Installing public key data
Jan 13 11:49:54 mds1 kernel: Loading keyring
Jan 13 11:49:54 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 11:49:54 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 11:49:54 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 11:49:54 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 11:49:54 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 11:49:54 mds1 kernel: io scheduler noop registered
Jan 13 11:49:54 mds1 kernel: io scheduler anticipatory registered
Jan 13 11:49:54 mds1 kernel: io scheduler deadline registered (default)
Jan 13 11:49:54 mds1 kernel: io scheduler cfq registered
Jan 13 11:49:54 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 11:49:54 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 11:49:54 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 11:49:54 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 11:49:54 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 11:49:54 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 11:49:54 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 11:49:54 mds1 kernel: ipmi message handler version 39.2
Jan 13 11:49:54 mds1 kernel: IPMI System Interface driver.
Jan 13 11:49:54 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 11:49:54 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 11:49:54 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 11:49:54 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 11:49:54 mds1 kernel: IPMI kcs interface initialized
Jan 13 11:49:54 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 11:49:54 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 11:49:54 mds1 kernel: ERST: Could not register with persistent store
Jan 13 11:49:54 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 11:49:54 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 11:49:54 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 11:49:54 mds1 kernel: Linux agpgart interface v0.103
Jan 13 11:49:54 mds1 kernel: crash memory driver: version 1.1
Jan 13 11:49:54 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 11:49:54 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 11:49:54 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 11:49:54 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 11:49:54 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 11:49:54 mds1 kernel: brd: module loaded
Jan 13 11:49:54 mds1 kernel: loop: module loaded
Jan 13 11:49:54 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 11:49:54 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 11:49:54 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 11:49:54 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 11:49:54 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 11:49:54 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 11:49:54 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 11:49:54 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 11:49:54 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 11:49:54 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 11:49:54 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 11:49:54 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 11:49:54 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 11:49:54 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 11:49:54 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 11:49:54 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 11:49:54 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 11:49:54 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 11:49:54 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 11:49:54 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 11:49:54 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 11:49:54 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 11:49:54 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 11:49:54 mds1 kernel: Switching to clocksource tsc
Jan 13 11:49:54 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 11:49:54 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 11:49:54 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 11:49:54 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 11:49:54 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 11:49:54 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 11:49:54 mds1 kernel: cpuidle: using governor ladder
Jan 13 11:49:54 mds1 kernel: cpuidle: using governor menu
Jan 13 11:49:54 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 11:49:54 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 11:49:54 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 11:49:54 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 11:49:54 mds1 kernel: TCP cubic registered
Jan 13 11:49:54 mds1 kernel: Initializing XFRM netlink socket
Jan 13 11:49:54 mds1 kernel: NET: Registered protocol family 17
Jan 13 11:49:54 mds1 kernel: registered taskstats version 1
Jan 13 11:49:54 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 16:49:34 UTC (1389631774)
Jan 13 11:49:54 mds1 kernel: Initalizing network drop monitor service
Jan 13 11:49:54 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 11:49:54 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 11:49:54 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 11:49:54 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 11:49:54 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 11:49:54 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 11:49:54 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 11:49:54 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 11:49:54 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 11:49:54 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 11:49:54 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 11:49:54 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 11:49:54 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 11:49:54 mds1 kernel: udev: starting version 147
Jan 13 11:49:54 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 11:49:54 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 11:49:54 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 11:49:54 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 11:49:54 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 11:49:54 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 11:49:54 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 11:49:54 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 11:49:54 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 11:49:54 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 11:49:54 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 11:49:54 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 11:49:54 mds1 kernel: megasas: FW now in Ready state
Jan 13 11:49:54 mds1 kernel: megasas:IOC Init cmd success
Jan 13 11:49:54 mds1 kernel: megasas: INIT adapter done
Jan 13 11:49:54 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 11:49:54 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 11:49:54 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 11:49:54 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 11:49:54 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 11:49:54 mds1 kernel: scsi1 : qla2xxx
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 11:49:54 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 11:49:54 mds1 kernel: scsi2 : qla2xxx
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 11:49:54 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 11:49:54 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 11:49:54 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 11:49:54 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 11:49:54 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 11:49:54 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 11:49:54 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 11:49:54 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 11:49:54 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2da7, mb[3]=7100, cmd=6a ****.
Jan 13 11:49:54 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2d89, mb[3]=a100, cmd=6a ****.
Jan 13 11:49:54 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 11:49:54 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 11:49:54 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 11:49:54 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 11:49:54 mds1 kernel: scsi3 : ahci
Jan 13 11:49:54 mds1 kernel: scsi4 : ahci
Jan 13 11:49:54 mds1 kernel: scsi5 : ahci
Jan 13 11:49:54 mds1 kernel: scsi6 : ahci
Jan 13 11:49:54 mds1 kernel: scsi7 : ahci
Jan 13 11:49:54 mds1 kernel: scsi8 : ahci
Jan 13 11:49:54 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 11:49:54 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 11:49:54 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 11:49:54 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 11:49:54 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 11:49:54 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 11:49:54 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 11:49:54 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 11:49:54 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 11:49:54 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 11:49:54 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 11:49:54 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sda:
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 11:49:54 mds1 kernel: sdb:
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 11:49:54 mds1 kernel: sdd:
Jan 13 11:49:54 mds1 kernel: sde: unknown partition table
Jan 13 11:49:54 mds1 kernel: unknown partition table
Jan 13 11:49:54 mds1 kernel: unknown partition table
Jan 13 11:49:54 mds1 kernel: sdf:
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sdc:
Jan 13 11:49:54 mds1 kernel: sdg:
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: unknown partition table
Jan 13 11:49:54 mds1 kernel: unknown partition table
Jan 13 11:49:54 mds1 kernel: unknown partition table
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sda1 sda2 sda3
Jan 13 11:49:54 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 11:49:54 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 11:49:54 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 11:49:54 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 11:49:54 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 11:49:54 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 11:49:54 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 11:49:54 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 11:49:54 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 11:49:54 mds1 kernel: type=1404 audit(1389631777.836:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 11:49:54 mds1 kernel: dracut: 
Jan 13 11:49:54 mds1 kernel: dracut: Switching root
Jan 13 11:49:54 mds1 kernel: udev: starting version 147
Jan 13 11:49:54 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 11:49:54 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 11:49:54 mds1 kernel: PTP clock support registered
Jan 13 11:49:54 mds1 kernel: dca service started, version 1.12.1
Jan 13 11:49:54 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 11:49:54 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 11:49:54 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 11:49:54 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 11:49:54 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 11:49:54 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 11:49:54 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 11:49:54 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 11:49:54 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 11:49:54 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 11:49:54 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 11:49:54 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 11:49:54 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 11:49:54 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 11:49:54 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 11:49:54 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 11:49:54 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 11:49:54 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 11:49:54 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 11:49:54 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 11:49:54 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 11:49:54 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 11:49:54 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 11:49:54 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 11:49:54 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 11:49:54 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 11:49:54 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 11:49:54 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 11:49:54 mds1 kernel: emc: device handler registered
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 11:49:54 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 11:49:54 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 11:49:54 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 11:49:54 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 11:49:54 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 11:49:54 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 11:49:54 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 11:49:54 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028f56000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 11:49:54 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028f56000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 11:49:54 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 11:49:54 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 11:49:54 mds1 kernel: NET: Registered protocol family 10
Jan 13 11:49:54 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 11:49:54 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 11:49:54 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 11:49:54 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 11:49:54 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 11:49:54 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 11:49:54 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 11:49:54 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 11:49:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 11:49:54 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 11:49:54 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 11:49:54 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 11:49:54 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 11:49:54 mds1 rpc.statd[3244]: Version 1.2.3 starting
Jan 13 11:49:54 mds1 sm-notify[3245]: Version 1.2.3 starting
Jan 13 11:49:55 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 11:49:55 mds1 kernel: RPC: Registered udp transport module.
Jan 13 11:49:55 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 11:49:55 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 11:49:55 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 11:49:55 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 11:49:55 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 11:49:55 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 11:49:59 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 11:49:59 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 11:50:03 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 11:50:03 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 11:50:03 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 11:50:03 mds1 kernel: LNet: Accept secure, port 988
Jan 13 11:50:04 mds1 cib[3407]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 attrd[3410]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 lrmd[3409]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 crmd[3412]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 stonith-ng[3408]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 pengine[3411]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 11:50:04 mds1 logger[3421]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 11:50:06 mds1 kdump: kexec: loaded kdump kernel
Jan 13 11:50:06 mds1 kdump: started up
Jan 13 11:50:06 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 11:50:06 mds1 kernel: Slow work thread pool: Starting up
Jan 13 11:50:06 mds1 kernel: Slow work thread pool: Ready
Jan 13 11:50:06 mds1 kernel: FS-Cache: Loaded
Jan 13 11:50:06 mds1 kernel: Registering the id_resolver key type
Jan 13 11:50:06 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 11:50:06 mds1 acpid: starting up
Jan 13 11:50:06 mds1 acpid: 1 rule loaded
Jan 13 11:50:06 mds1 acpid: waiting for events: event logging is off
Jan 13 11:50:07 mds1 acpid: client connected from 3637[68:68]
Jan 13 11:50:07 mds1 acpid: 1 client rule loaded
Jan 13 11:50:07 mds1 kernel: ipmi device interface
Jan 13 11:50:08 mds1 automount[3676]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 11:50:09 mds1 abrtd: Init complete, entering main loop
Jan 13 11:50:09 mds1 cimserver[3827]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 11:50:15 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 11:50:17 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 11:50:33 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 11:50:33 mds1 kernel: 
Jan 13 11:50:33 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 11:50:33 mds1 kernel: 
Jan 13 11:50:33 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 11:50:33 mds1 kernel: 
Jan 13 11:51:15 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 13 11:51:15 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 11:51:16 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 13 11:51:16 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 13 11:51:16 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 11:51:16 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 11:51:17 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 11:51:17 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:51:17 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 11:51:17 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 11:51:17 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:51:17 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 11:51:17 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 11:51:17 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 11:51:18 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 6 clients reconnect
Jan 13 11:51:22 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389631877/real 0]  req@ffff880821c8c800 x1457134557463000/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389631882 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:51:24 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 8 clients reconnect
Jan 13 11:51:42 mds1 kernel: LustreError: 11-0: emc-OST000a-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -11.
Jan 13 11:51:42 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 11:51:48 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389631902/real 0]  req@ffff880fe0722000 x1457134557463316/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389631908 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:51:48 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 13 11:52:07 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client emc-MDT0001-mdtlov_UUID (at 0@lo), waiting for all 6 known clients (5 recovered, 0 in progress, and 0 evicted) to recover in 4:10
Jan 13 11:52:07 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -16.
Jan 13 11:52:07 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 11:52:32 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client emc-MDT0001-mdtlov_UUID (at 0@lo), waiting for all 6 known clients (5 recovered, 0 in progress, and 0 evicted) to recover in 3:45
Jan 13 11:52:32 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -16.
Jan 13 11:52:48 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 1:30, of 6 clients 6 recovered and 0 were evicted.
Jan 13 11:53:08 mds1 kernel: Lustre: emc-OST0002-osc-MDT0001: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 13 11:53:33 mds1 kernel: Lustre: emc-OST000b-osc-MDT0001: Connection restored to emc-OST000b (at 192.168.1.46@o2ib)
Jan 13 11:53:33 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 11:53:36 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 13 11:53:36 mds1 kernel: Lustre: emc-MDT0001: Not available for connect from 0@lo (stopping)
Jan 13 11:53:36 mds1 kernel: Lustre: 4533:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 13 11:53:36 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 2:12, of 8 clients 0 recovered and 0 were evicted.
Jan 13 11:53:36 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_disconnect failed with -107.
Jan 13 11:53:36 mds1 kernel: Lustre: setting import emc-OST0001_UUID INACTIVE by administrator request
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(osp_dev.c:338:osp_disconnect()) emc-OST0001-osc-MDT0001: can't disconnect: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 1: cmd 0xcf031: rc = -5
Jan 13 11:53:36 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 192.168.1.46@o2ib (stopping)
Jan 13 11:53:36 mds1 kernel: Lustre: setting import emc-OST0006_UUID INACTIVE by administrator request
Jan 13 11:53:36 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(osp_dev.c:338:osp_disconnect()) emc-OST0006-osc-MDT0001: can't disconnect: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(osp_dev.c:338:osp_disconnect()) Skipped 1 previous similar message
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 6: cmd 0xcf031: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 5145:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) Skipped 1 previous similar message
Jan 13 11:53:36 mds1 kernel: LustreError: 4483:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST000c-osc-MDT0001: can't precreate: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 4509:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST000d-osc-MDT0001: cannot precreate objects: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 4483:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 2 previous similar messages
Jan 13 11:53:36 mds1 kernel: LustreError: 4525:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0015-osc-MDT0001: can't precreate: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 4525:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 2 previous similar messages
Jan 13 11:53:36 mds1 kernel: LustreError: 4529:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0017-osc-MDT0001: cannot precreate objects: rc = -5
Jan 13 11:53:36 mds1 kernel: LustreError: 4529:0:(osp_precreate.c:986:osp_precreate_thread()) Skipped 6 previous similar messages
Jan 13 11:53:37 mds1 kernel: Lustre: server umount MGS complete
Jan 13 11:53:43 mds1 kernel: Lustre: 5144:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389632017/real 1389632017]  req@ffff880fe071f400 x1457134557465200/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389632023 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:53:43 mds1 kernel: Lustre: 5144:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 11:53:43 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 11:53:43 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 11:53:46 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 11:53:49 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.56@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:53:51 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 11:53:51 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389632026/real 1389632026]  req@ffff8810174a1c00 x1457134557465204/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389632031 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:53:51 mds1 kernel: LustreError: 5254:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 13 11:53:51 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 11:53:51 mds1 kernel: LustreError: 5452:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 11:53:51 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 11:53:51 mds1 kernel: LustreError: Skipped 14 previous similar messages
Jan 13 11:53:56 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389632031/real 0]  req@ffff880fe0947800 x1457134557465296/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389632036 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 11:53:58 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe075ec00 x1457134557465564/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:00 mds1 kernel: LustreError: 5302:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe0712800 x1457134557465568/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:04 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe075ec00 x1457134557465572/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:04 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 11:54:04 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 11:54:06 mds1 kernel: LustreError: 5302:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe0712800 x1457134557465576/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:10 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe075ec00 x1457134557465580/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:10 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 11:54:16 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe075ec00 x1457134557465588/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:16 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 11:54:27 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880fe075ec00 x1457134557465592/t0(0) o253->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 13 11:54:27 mds1 kernel: LustreError: 5254:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 13 11:54:35 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 11:54:44 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 7 clients reconnect
Jan 13 11:55:00 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 13 11:55:00 mds1 kernel: LustreError: Skipped 7 previous similar messages
Jan 13 11:56:15 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 13 11:56:15 mds1 kernel: LustreError: Skipped 13 previous similar messages
Jan 13 11:57:33 mds1 kernel: INFO: task tgt_recov:5584 blocked for more than 120 seconds.
Jan 13 11:57:33 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 13 11:57:33 mds1 kernel: tgt_recov     D 0000000000000006     0  5584      2 0x00000080
Jan 13 11:57:33 mds1 kernel: ffff880fe513fda0 0000000000000046 0000000000000000 ffff88085c456740
Jan 13 11:57:33 mds1 kernel: ffff880fe513fd30 ffffffff81055f96 ffff880fe513fd40 ffff88100595e040
Jan 13 11:57:33 mds1 kernel: ffff881017c4faf8 ffff880fe513ffd8 000000000000fb88 ffff881017c4faf8
Jan 13 11:57:33 mds1 kernel: Call Trace:
Jan 13 11:57:33 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 13 11:57:33 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 11:57:33 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 13 11:57:33 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 13 11:57:33 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 13 11:57:33 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 13 11:59:33 mds1 kernel: INFO: task tgt_recov:5584 blocked for more than 120 seconds.
Jan 13 11:59:33 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 13 11:59:33 mds1 kernel: tgt_recov     D 0000000000000006     0  5584      2 0x00000080
Jan 13 11:59:33 mds1 kernel: ffff880fe513fda0 0000000000000046 0000000000000000 ffff88085c456740
Jan 13 11:59:33 mds1 kernel: ffff880fe513fd30 ffffffff81055f96 ffff880fe513fd40 ffff88100595e040
Jan 13 11:59:33 mds1 kernel: ffff881017c4faf8 ffff880fe513ffd8 000000000000fb88 ffff881017c4faf8
Jan 13 11:59:33 mds1 kernel: Call Trace:
Jan 13 11:59:33 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 13 11:59:33 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 11:59:33 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 13 11:59:33 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 13 11:59:33 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 13 11:59:33 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 13 11:59:44 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 13 11:59:44 mds1 kernel: Lustre: emc-MDT0000: disconnecting 6 stale clients
Jan 13 11:59:44 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 5:00, of 7 clients 1 recovered and 6 were evicted.
Jan 13 12:00:00 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 13 12:00:00 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 12:07:36 mds1 kernel: Lustre: 3385:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389632849/real 0]  req@ffff880821c76400 x1457134557476680/t0(0) o13->emc-OST0004-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389632856 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 12:07:36 mds1 kernel: Lustre: 3385:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 12:07:36 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection to emc-OST0004 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:07:36 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:07:36 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 13 12:07:36 mds1 kernel: Lustre: 3391:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389632849/real 0]  req@ffff880831b43000 x1457134557476608/t0(0) o13->emc-OST0015-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389632856 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 12:07:36 mds1 kernel: Lustre: 3391:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jan 13 12:07:36 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:07:36 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 13 12:07:42 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389632856/real 0]  req@ffff8808323eb000 x1457134557476788/t0(0) o8->emc-OST0003-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389632862 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:07:42 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
Jan 13 12:08:08 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389632882/real 1389632882]  req@ffff8808220c8800 x1457134557476816/t0(0) o8->emc-OST0008-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389632888 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:08:08 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:08:36 mds1 kernel: Lustre: MGS: Client a426c3d7-5c5e-bd7c-8c47-73aa55bd6cc5 (at 192.168.1.44@o2ib) reconnecting
Jan 13 12:08:37 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:08:37 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000a_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:08:37 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0009_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:08:37 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 13 12:08:38 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389632907/real 0]  req@ffff880822169800 x1457134557477004/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389632918 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:08:38 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:08:39 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000c_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:08:39 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 13 12:08:43 mds1 kernel: LustreError: 3380:0:(import.c:983:ptlrpc_connect_interpret()) emc-OST0005_UUID went back in time (transno 498216206340 was previously committed, server now claims 498216206338)!  See https://bugzilla.lustre.org/show_bug.cgi?id=9646
Jan 13 12:08:43 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection restored to emc-OST000a (at 192.168.1.44@o2ib)
Jan 13 12:08:43 mds1 kernel: LustreError: 3380:0:(import.c:983:ptlrpc_connect_interpret()) emc-OST000b_UUID went back in time (transno 369367187460 was previously committed, server now claims 369367187458)!  See https://bugzilla.lustre.org/show_bug.cgi?id=9646
Jan 13 12:09:30 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:09:30 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 13 12:10:20 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:10:20 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:11:10 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:11:10 mds1 kernel: Lustre: Skipped 13 previous similar messages
Jan 13 12:11:22 mds1 kernel: Lustre: MGS: haven't heard from client 520ce816-1b62-68f6-3422-f55ce6a57f80 (at 192.168.1.46@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff8810174d2400, cur 1389633082 expire 1389632932 last 1389632834
Jan 13 12:11:35 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0003_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:11:35 mds1 kernel: Lustre: Skipped 18 previous similar messages
Jan 13 12:11:47 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST000f_UUID (at 192.168.1.46@o2ib) in 273 seconds. I think it's dead, and I am evicting it. exp ffff880fe0fc5c00, cur 1389633107 expire 1389632957 last 1389632834
Jan 13 12:15:30 mds1 kernel: Lustre: 3386:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389633318/real 0]  req@ffff880821e3ac00 x1457134557485220/t0(0) o13->emc-OST0012-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389633330 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 12:15:30 mds1 kernel: Lustre: 3386:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:15:30 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:15:30 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 13 12:16:07 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389633356/real 1389633356]  req@ffff88082233e800 x1457134557485392/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389633367 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:16:07 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 12:16:31 mds1 kernel: Lustre: MGS: Client 2ab5e82c-d985-34aa-9269-e97f7d694e1f (at 192.168.1.46@o2ib) reconnecting
Jan 13 12:16:31 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0010_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 13 12:16:31 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 13 12:16:41 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection restored to emc-OST000a (at 192.168.1.46@o2ib)
Jan 13 12:16:41 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:18:13 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0011_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 13 12:18:13 mds1 kernel: Lustre: Skipped 49 previous similar messages
Jan 13 12:18:53 mds1 kernel: Lustre: MGS: haven't heard from client a426c3d7-5c5e-bd7c-8c47-73aa55bd6cc5 (at 192.168.1.44@o2ib) in 238 seconds. I think it's dead, and I am evicting it. exp ffff880fe3a2e000, cur 1389633533 expire 1389633383 last 1389633295
Jan 13 12:18:53 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:26:58 mds1 kernel: Lustre: 3388:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634011/real 0]  req@ffff880824617800 x1457134557498020/t0(0) o13->emc-OST0007-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389634018 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 12:26:58 mds1 kernel: Lustre: 3388:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 12:26:58 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection to emc-OST0007 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:26:58 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:26:58 mds1 kernel: Lustre: emc-OST000d-osc-MDT0000: Connection to emc-OST000d (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:26:58 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 13 12:27:04 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634018/real 0]  req@ffff880821c10400 x1457134557498196/t0(0) o8->emc-OST0014-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389634024 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:27:04 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:27:29 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389634043/real 1389634043]  req@ffff880822546c00 x1457134557498212/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389634049 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:27:29 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:27:59 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634068/real 0]  req@ffff88082276a000 x1457134557498408/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389634079 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:27:59 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 12:28:05 mds1 kernel: Lustre: MGS: Client 40c11fdf-55c0-7da3-952f-3c295734bbb1 (at 192.168.1.44@o2ib) reconnecting
Jan 13 12:28:05 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0008_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 13 12:28:05 mds1 kernel: Lustre: Skipped 13 previous similar messages
Jan 13 12:28:14 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection restored to emc-OST0012 (at 192.168.1.44@o2ib)
Jan 13 12:28:14 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:30:36 mds1 kernel: Lustre: MGS: haven't heard from client 2ab5e82c-d985-34aa-9269-e97f7d694e1f (at 192.168.1.46@o2ib) in 243 seconds. I think it's dead, and I am evicting it. exp ffff88083226a800, cur 1389634236 expire 1389634086 last 1389633993
Jan 13 12:30:36 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 13 12:30:43 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0010_UUID (at 192.168.1.46@o2ib) in 250 seconds. I think it's dead, and I am evicting it. exp ffff880ff3da9000, cur 1389634243 expire 1389634093 last 1389633993
Jan 13 12:34:54 mds1 kernel: LustreError: 11-0: emc-OST0012-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 13 12:34:54 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection to emc-OST0003 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:34:54 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 13 12:34:54 mds1 kernel: LustreError: Skipped 17 previous similar messages
Jan 13 12:35:05 mds1 kernel: Lustre: MGS: Client 31a22afc-5160-e96b-b50f-5b7339b1544b (at 192.168.1.46@o2ib) reconnecting
Jan 13 12:35:15 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 13 12:35:15 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 12:40:23 mds1 kernel: Lustre: 3388:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634815/real 0]  req@ffff880822b6d800 x1457134557513836/t0(0) o13->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389634823 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 12:40:23 mds1 kernel: Lustre: 3388:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 13 12:40:23 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection to emc-OST0016 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:40:23 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 13 12:40:26 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 12:40:26 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 13 12:40:30 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634823/real 0]  req@ffff880822f5b000 x1457134557514008/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389634830 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:40:30 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 13 12:40:38 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:40:42 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:40:51 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:40:52 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.51@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:40:57 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.52@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:41:17 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.53@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 12:41:27 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389634876/real 0]  req@ffff88082296cc00 x1457134557514608/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389634887 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 12:41:27 mds1 kernel: Lustre: 3380:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 12:41:27 mds1 init: tty (/dev/tty1) main process (3852) killed by TERM signal
Jan 13 12:41:27 mds1 init: tty (/dev/tty2) main process (3854) killed by TERM signal
Jan 13 12:41:27 mds1 init: tty (/dev/tty3) main process (3856) killed by TERM signal
Jan 13 12:41:27 mds1 init: tty (/dev/tty4) main process (3858) killed by TERM signal
Jan 13 12:41:27 mds1 init: tty (/dev/tty5) main process (3860) killed by TERM signal
Jan 13 12:41:27 mds1 init: tty (/dev/tty6) main process (3862) killed by TERM signal
Jan 13 12:41:32 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 14:23:18 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 14:23:18 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3167" x-info="http://www.rsyslog.com"] start
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 14:23:18 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 14:23:18 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 14:23:18 mds1 kernel: KERNEL supported cpus:
Jan 13 14:23:18 mds1 kernel:  Intel GenuineIntel
Jan 13 14:23:18 mds1 kernel:  AMD AuthenticAMD
Jan 13 14:23:18 mds1 kernel:  Centaur CentaurHauls
Jan 13 14:23:18 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 14:23:18 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 14:23:18 mds1 kernel: DMI 2.6 present.
Jan 13 14:23:18 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 14:23:18 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 14:23:18 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 14:23:18 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 14:23:18 mds1 kernel: Using GB pages for direct mapping
Jan 13 14:23:18 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 14:23:18 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 14:23:18 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 14:23:18 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 14:23:18 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 14:23:18 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 14:23:18 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 14:23:18 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 14:23:18 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 14:23:18 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 14:23:18 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 14:23:18 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 14:23:18 mds1 kernel: Setting APIC routing to flat.
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 14:23:18 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 14:23:18 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 14:23:18 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 14:23:18 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 14:23:18 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 14:23:18 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 14:23:18 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 14:23:18 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 14:23:18 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 14:23:18 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 14:23:18 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 14:23:18 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 14:23:18 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 14:23:18 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 14:23:18 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 14:23:18 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 14:23:18 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 14:23:18 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 14:23:18 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 14:23:18 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 14:23:18 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 14:23:18 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 14:23:18 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 14:23:18 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 14:23:18 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 14:23:18 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 14:23:18 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 14:23:18 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 14:23:18 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 14:23:18 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 14:23:18 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 14:23:18 mds1 kernel: Zone PFN ranges:
Jan 13 14:23:18 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 14:23:18 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 14:23:18 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 14:23:18 mds1 kernel: Movable zone start PFN for each node
Jan 13 14:23:18 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 14:23:18 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 14:23:18 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 14:23:18 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 14:23:18 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 14:23:18 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 14:23:18 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 14:23:18 mds1 kernel: Setting APIC routing to flat.
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 14:23:18 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 14:23:18 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 14:23:18 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 14:23:18 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 14:23:18 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 14:23:18 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 14:23:18 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 14:23:18 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 14:23:18 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 14:23:18 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 14:23:18 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 14:23:18 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 14:23:18 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 14:23:18 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 14:23:18 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 14:23:18 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 14:23:18 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 14:23:18 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 14:23:18 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 14:23:18 mds1 kernel: Policy zone: Normal
Jan 13 14:23:18 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 14:23:18 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 14:23:18 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 14:23:18 mds1 kernel: Checking aperture...
Jan 13 14:23:18 mds1 kernel: No AGP bridge found
Jan 13 14:23:18 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 14:23:18 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 14:23:18 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 14:23:18 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 14:23:18 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 14:23:18 mds1 kernel: Hierarchical RCU implementation.
Jan 13 14:23:18 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 14:23:18 mds1 kernel: Extended CMOS year: 2000
Jan 13 14:23:18 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 14:23:18 mds1 kernel: console [tty0] enabled
Jan 13 14:23:18 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 14:23:18 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 14:23:18 mds1 kernel: Fast TSC calibration using PIT
Jan 13 14:23:18 mds1 kernel: Detected 1995.142 MHz processor.
Jan 13 14:23:18 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.28 BogoMIPS (lpj=1995142)
Jan 13 14:23:18 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 14:23:18 mds1 kernel: Security Framework initialized
Jan 13 14:23:18 mds1 kernel: SELinux:  Initializing.
Jan 13 14:23:18 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 14:23:18 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 14:23:18 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys ns
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys memory
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys devices
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 14:23:18 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 14:23:18 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 14:23:18 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 14:23:18 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 14:23:18 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 14:23:18 mds1 kernel: using mwait in idle threads.
Jan 13 14:23:18 mds1 kernel: ACPI: Core revision 20090903
Jan 13 14:23:18 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 14:23:18 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 14:23:18 mds1 kernel: dmar: Host address width 46
Jan 13 14:23:18 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 14:23:18 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 14:23:18 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 14:23:18 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 14:23:18 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 14:23:18 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 14:23:18 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 14:23:18 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 14:23:18 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 14:23:18 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 14:23:18 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 14:23:18 mds1 kernel: Enabling x2apic
Jan 13 14:23:18 mds1 kernel: Enabled x2apic
Jan 13 14:23:18 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 14:23:18 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 14:23:18 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 14:23:18 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 14:23:18 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 14:23:18 mds1 kernel: ... version:                3
Jan 13 14:23:18 mds1 kernel: ... bit width:              48
Jan 13 14:23:18 mds1 kernel: ... generic registers:      8
Jan 13 14:23:18 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 14:23:18 mds1 kernel: ... max period:             000000007fffffff
Jan 13 14:23:18 mds1 kernel: ... fixed-purpose events:   3
Jan 13 14:23:18 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 14:23:18 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 14:23:18 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 14:23:18 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 14:23:18 mds1 kernel: Brought up 12 CPUs
Jan 13 14:23:18 mds1 kernel: Total of 12 processors activated (47880.76 BogoMIPS).
Jan 13 14:23:18 mds1 kernel: devtmpfs: initialized
Jan 13 14:23:18 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 14:23:18 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 14:23:18 mds1 kernel: regulator: core version 0.5
Jan 13 14:23:18 mds1 kernel: NET: Registered protocol family 16
Jan 13 14:23:18 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 14:23:18 mds1 kernel: ACPI: bus type pci registered
Jan 13 14:23:18 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 14:23:18 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 14:23:18 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 14:23:18 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 14:23:18 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 14:23:18 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 14:23:18 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 14:23:18 mds1 kernel: ACPI: Interpreter enabled
Jan 13 14:23:18 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 14:23:18 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 14:23:18 mds1 kernel: ACPI: No dock devices found.
Jan 13 14:23:18 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 14:23:18 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 14:23:18 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 14:23:18 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 14:23:18 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 14:23:18 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 14:23:18 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 14:23:18 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 14:23:18 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 14:23:18 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 14:23:18 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 14:23:18 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 14:23:18 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 14:23:18 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 14:23:18 mds1 kernel: vgaarb: loaded
Jan 13 14:23:18 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 14:23:18 mds1 kernel: SCSI subsystem initialized
Jan 13 14:23:18 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 14:23:18 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 14:23:18 mds1 kernel: usbcore: registered new device driver usb
Jan 13 14:23:18 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 14:23:18 mds1 kernel: NetLabel: Initializing
Jan 13 14:23:18 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 14:23:18 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 14:23:18 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 14:23:18 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 14:23:18 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 14:23:18 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 14:23:18 mds1 kernel: Switching to clocksource hpet
Jan 13 14:23:18 mds1 kernel: pnp: PnP ACPI init
Jan 13 14:23:18 mds1 kernel: ACPI: bus type pnp registered
Jan 13 14:23:18 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 14:23:18 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 14:23:18 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 14:23:18 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 14:23:18 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 14:23:18 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 14:23:18 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 14:23:18 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 14:23:18 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 14:23:18 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 14:23:18 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 14:23:18 mds1 kernel: NET: Registered protocol family 2
Jan 13 14:23:18 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 14:23:18 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 14:23:18 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 14:23:18 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 14:23:18 mds1 kernel: TCP reno registered
Jan 13 14:23:18 mds1 kernel: NET: Registered protocol family 1
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 14:23:18 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 14:23:18 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 14:23:18 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 14:23:18 mds1 kernel: dmar: Unsupported device scope
Jan 13 14:23:18 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 14:23:18 mds1 kernel: type=2000 audit(1389640976.053:1): initialized
Jan 13 14:23:18 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 14:23:18 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 14:23:18 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 14:23:18 mds1 kernel: msgmni has been set to 32768
Jan 13 14:23:18 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 14:23:18 mds1 kernel: ksign: Installing public key data
Jan 13 14:23:18 mds1 kernel: Loading keyring
Jan 13 14:23:18 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 14:23:18 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 14:23:18 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 14:23:18 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 14:23:18 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 14:23:18 mds1 kernel: io scheduler noop registered
Jan 13 14:23:18 mds1 kernel: io scheduler anticipatory registered
Jan 13 14:23:18 mds1 kernel: io scheduler deadline registered (default)
Jan 13 14:23:18 mds1 kernel: io scheduler cfq registered
Jan 13 14:23:18 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 14:23:18 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 14:23:18 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 14:23:18 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 14:23:18 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 14:23:18 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 14:23:18 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 14:23:18 mds1 kernel: ipmi message handler version 39.2
Jan 13 14:23:18 mds1 kernel: IPMI System Interface driver.
Jan 13 14:23:18 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 14:23:18 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 14:23:18 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 14:23:18 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 14:23:18 mds1 kernel: IPMI kcs interface initialized
Jan 13 14:23:18 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 14:23:18 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 14:23:18 mds1 kernel: ERST: Could not register with persistent store
Jan 13 14:23:18 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 14:23:18 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 14:23:18 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 14:23:18 mds1 kernel: Linux agpgart interface v0.103
Jan 13 14:23:18 mds1 kernel: crash memory driver: version 1.1
Jan 13 14:23:18 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 14:23:18 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 14:23:18 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 14:23:18 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 14:23:18 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 14:23:18 mds1 kernel: brd: module loaded
Jan 13 14:23:18 mds1 kernel: loop: module loaded
Jan 13 14:23:18 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 14:23:18 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 14:23:18 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 14:23:18 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 14:23:18 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 14:23:18 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 14:23:18 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 14:23:18 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 14:23:18 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 14:23:18 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 14:23:18 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 14:23:18 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 14:23:18 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 14:23:18 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 14:23:18 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 14:23:18 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 14:23:18 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 14:23:18 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 14:23:18 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 14:23:18 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 14:23:18 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 14:23:18 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 14:23:18 mds1 kernel: Refined TSC clocksource calibration: 1995.191 MHz.
Jan 13 14:23:18 mds1 kernel: Switching to clocksource tsc
Jan 13 14:23:18 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 14:23:18 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 14:23:18 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 14:23:18 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 14:23:18 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 14:23:18 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 14:23:18 mds1 kernel: cpuidle: using governor ladder
Jan 13 14:23:18 mds1 kernel: cpuidle: using governor menu
Jan 13 14:23:18 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 14:23:18 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 14:23:18 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 14:23:18 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 14:23:18 mds1 kernel: TCP cubic registered
Jan 13 14:23:18 mds1 kernel: Initializing XFRM netlink socket
Jan 13 14:23:18 mds1 kernel: NET: Registered protocol family 17
Jan 13 14:23:18 mds1 kernel: registered taskstats version 1
Jan 13 14:23:18 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 19:22:58 UTC (1389640978)
Jan 13 14:23:18 mds1 kernel: Initalizing network drop monitor service
Jan 13 14:23:18 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 14:23:18 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 14:23:18 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 14:23:18 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 14:23:18 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 14:23:18 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 14:23:18 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 14:23:18 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 14:23:18 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 14:23:18 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 14:23:18 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 14:23:18 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 14:23:18 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 14:23:18 mds1 kernel: udev: starting version 147
Jan 13 14:23:18 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 14:23:18 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 14:23:18 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 14:23:18 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 14:23:18 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 14:23:18 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 14:23:18 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 14:23:18 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 14:23:18 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 14:23:18 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 14:23:18 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 14:23:18 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 14:23:18 mds1 kernel: megasas: FW now in Ready state
Jan 13 14:23:18 mds1 kernel: megasas:IOC Init cmd success
Jan 13 14:23:18 mds1 kernel: megasas: INIT adapter done
Jan 13 14:23:18 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 14:23:18 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 14:23:18 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 14:23:18 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 14:23:18 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 14:23:18 mds1 kernel: scsi1 : qla2xxx
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 14:23:18 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 14:23:18 mds1 kernel: scsi2 : qla2xxx
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 14:23:18 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 14:23:18 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 14:23:18 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 14:23:18 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 14:23:18 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 14:23:18 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 14:23:18 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 14:23:18 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 14:23:18 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dc4, mb[3]=4100, cmd=6a ****.
Jan 13 14:23:18 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2de0, mb[3]=e100, cmd=6a ****.
Jan 13 14:23:18 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 14:23:18 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 14:23:18 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 14:23:18 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 14:23:18 mds1 kernel: scsi3 : ahci
Jan 13 14:23:18 mds1 kernel: scsi4 : ahci
Jan 13 14:23:18 mds1 kernel: scsi5 : ahci
Jan 13 14:23:18 mds1 kernel: scsi6 : ahci
Jan 13 14:23:18 mds1 kernel: scsi7 : ahci
Jan 13 14:23:18 mds1 kernel: scsi8 : ahci
Jan 13 14:23:18 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 14:23:18 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 14:23:18 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 14:23:18 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 14:23:18 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 14:23:18 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 14:23:18 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 14:23:18 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 14:23:18 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 14:23:18 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 14:23:18 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 14:23:18 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sda:
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sdb:
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:23:18 mds1 kernel: sdc: unknown partition table
Jan 13 14:23:18 mds1 kernel: unknown partition table
Jan 13 14:23:18 mds1 kernel: sdd:
Jan 13 14:23:18 mds1 kernel: sde:
Jan 13 14:23:18 mds1 kernel: sdf:
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: sdg:
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: unknown partition table
Jan 13 14:23:18 mds1 kernel: unknown partition table
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: unknown partition table
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: unknown partition table
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: sda1 sda2 sda3
Jan 13 14:23:18 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 14:23:18 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 14:23:18 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 14:23:18 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 13 14:23:18 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 13 14:23:18 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 13 14:23:18 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 14:23:18 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 14:23:18 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 14:23:18 mds1 kernel: type=1404 audit(1389640981.547:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 14:23:18 mds1 kernel: dracut: 
Jan 13 14:23:18 mds1 kernel: dracut: Switching root
Jan 13 14:23:18 mds1 kernel: udev: starting version 147
Jan 13 14:23:18 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 14:23:18 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 14:23:18 mds1 kernel: PTP clock support registered
Jan 13 14:23:18 mds1 kernel: dca service started, version 1.12.1
Jan 13 14:23:18 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 14:23:18 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 14:23:18 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 14:23:18 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 14:23:18 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 14:23:18 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 14:23:18 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 14:23:18 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 14:23:18 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 14:23:18 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 14:23:18 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 14:23:18 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 14:23:18 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 14:23:18 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 14:23:18 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 14:23:18 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 14:23:18 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 14:23:18 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 14:23:18 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 14:23:18 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 14:23:18 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 14:23:18 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 14:23:18 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 14:23:18 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 14:23:18 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 14:23:18 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 14:23:18 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 14:23:18 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 14:23:18 mds1 kernel: emc: device handler registered
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 14:23:18 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 14:23:18 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 14:23:18 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 14:23:18 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 14:23:18 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 14:23:18 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 14:23:18 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 14:23:18 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10281a8000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 14:23:18 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10281a8000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 14:23:18 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 14:23:18 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 14:23:18 mds1 kernel: NET: Registered protocol family 10
Jan 13 14:23:18 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 14:23:18 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 14:23:18 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 14:23:18 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 14:23:18 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 14:23:18 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 14:23:18 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 14:23:18 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 14:23:18 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 14:23:18 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 14:23:18 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 14:23:18 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 14:23:18 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 14:23:18 mds1 rpc.statd[3262]: Version 1.2.3 starting
Jan 13 14:23:18 mds1 sm-notify[3263]: Version 1.2.3 starting
Jan 13 14:23:18 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 14:23:18 mds1 kernel: RPC: Registered udp transport module.
Jan 13 14:23:18 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 14:23:18 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 14:23:18 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 14:23:18 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 14:23:18 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 14:23:18 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 14:23:22 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 14:23:23 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 14:23:27 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 14:23:27 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 14:23:27 mds1 kernel: LNet: Added LNI 10.245.60.40@tcp [8/256/0/180]
Jan 13 14:23:27 mds1 kernel: LNet: Accept secure, port 988
Jan 13 14:23:28 mds1 lrmd[3427]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 attrd[3428]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 cib[3425]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 crmd[3430]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 stonith-ng[3426]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 pengine[3429]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 14:23:28 mds1 logger[3439]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 14:23:29 mds1 kdump: kexec: loaded kdump kernel
Jan 13 14:23:29 mds1 kdump: started up
Jan 13 14:23:30 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 14:23:30 mds1 kernel: Slow work thread pool: Starting up
Jan 13 14:23:30 mds1 kernel: Slow work thread pool: Ready
Jan 13 14:23:30 mds1 kernel: FS-Cache: Loaded
Jan 13 14:23:30 mds1 kernel: Registering the id_resolver key type
Jan 13 14:23:30 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 14:23:30 mds1 acpid: starting up
Jan 13 14:23:30 mds1 acpid: 1 rule loaded
Jan 13 14:23:30 mds1 acpid: waiting for events: event logging is off
Jan 13 14:23:31 mds1 acpid: client connected from 3656[68:68]
Jan 13 14:23:31 mds1 acpid: 1 client rule loaded
Jan 13 14:23:31 mds1 kernel: ipmi device interface
Jan 13 14:23:32 mds1 automount[3695]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 14:23:33 mds1 abrtd: Init complete, entering main loop
Jan 13 14:23:33 mds1 cimserver[3846]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 14:23:39 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 14:23:41 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 14:24:11 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 14:24:11 mds1 kernel: 
Jan 13 14:24:11 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 14:24:11 mds1 kernel: 
Jan 13 14:24:11 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 13 14:24:11 mds1 kernel: 
Jan 13 14:24:25 mds1 cibadmin[4325]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 13 14:24:53 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 13 14:24:54 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 13 14:24:54 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 13 14:24:54 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 14:24:54 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 14:24:54 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 14:24:58 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.53@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 14:25:00 mds1 kernel: Lustre: 3398:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389641095/real 0]  req@ffff880fe1c73c00 x1457144208556040/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389641100 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 14:25:00 mds1 kernel: LustreError: 4224:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 13 14:25:00 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 14:25:00 mds1 kernel: LustreError: 4477:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 14:25:00 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 14:25:00 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 14:25:00 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 14:25:00 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -19.
Jan 13 14:25:00 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 14:25:00 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 14:25:00 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 14:25:00 mds1 kernel: LustreError: 4545:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 13 14:25:02 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 7 clients reconnect
Jan 13 14:25:05 mds1 kernel: Lustre: 3398:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389641100/real 0]  req@ffff880fdff3c000 x1457144208556420/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389641105 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 14:25:31 mds1 kernel: Lustre: 3398:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389641125/real 0]  req@ffff880fdcfb1400 x1457144208556844/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389641131 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 14:27:16 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 14:27:16 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_disconnect failed with -19.
Jan 13 14:27:16 mds1 kernel: LustreError: Skipped 9 previous similar messages
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(lwp_dev.c:176:lwp_disconnect()) emc-MDT0000-lwp-MDT0001: can't disconnect: rc = -19
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0001: failed to disconnect lwp. (rc=-19)
Jan 13 14:27:16 mds1 kernel: LustreError: 5120:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0000: Aborting recovery
Jan 13 14:27:16 mds1 kernel: Lustre: 4598:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 13 14:27:16 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 2:14, of 7 clients 0 recovered and 0 were evicted.
Jan 13 14:27:16 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 13 14:27:16 mds1 kernel: Lustre: 4544:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 13 14:27:16 mds1 kernel: Lustre: 4544:0:(ldlm_lib.c:1798:target_recovery_overseer()) Skipped 2 previous similar messages
Jan 13 14:27:16 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 2:14, of 7 clients 0 recovered and 0 were evicted.
Jan 13 14:27:16 mds1 kernel: Lustre: emc-MDT0001: Not available for connect from 0@lo (stopping)
Jan 13 14:27:16 mds1 kernel: LustreError: 4562:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4566:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST0002-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4566:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message
Jan 13 14:27:16 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 0@lo (stopping)
Jan 13 14:27:16 mds1 kernel: Lustre: setting import emc-OST0000_UUID INACTIVE by administrator request
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(osp_dev.c:338:osp_disconnect()) emc-OST0000-osc-MDT0001: can't disconnect: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031: rc = -5
Jan 13 14:27:16 mds1 kernel: Lustre: setting import emc-OST0001_UUID INACTIVE by administrator request
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(osp_dev.c:338:osp_disconnect()) emc-OST0001-osc-MDT0001: can't disconnect: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 5122:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 1: cmd 0xcf031: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4515:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0005-osc-MDT0001: can't precreate: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4515:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0005-osc-MDT0001: cannot precreate objects: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4515:0:(osp_precreate.c:986:osp_precreate_thread()) Skipped 2 previous similar messages
Jan 13 14:27:16 mds1 kernel: LustreError: 4491:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST000c-osc-MDT0001: can't precreate: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4517:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST000d-osc-MDT0001: cannot precreate objects: rc = -5
Jan 13 14:27:16 mds1 kernel: LustreError: 4517:0:(osp_precreate.c:986:osp_precreate_thread()) Skipped 3 previous similar messages
Jan 13 14:27:16 mds1 kernel: LustreError: 4491:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 9 previous similar messages
Jan 13 14:27:16 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 13 14:27:26 mds1 kernel: Lustre: 5118:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389641236/real 1389641236]  req@ffff8810305e5400 x1457144208558472/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389641246 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 14:27:26 mds1 kernel: Lustre: 5118:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 13 14:27:26 mds1 kernel: Lustre: server umount MGS complete
Jan 13 14:27:26 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 14:29:09 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 14:29:11 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 14:29:11 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 14:29:11 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 13 14:29:11 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 14:29:11 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 14:29:23 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 7 clients reconnect
Jan 13 14:29:23 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 14:29:36 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 14:29:36 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 14:29:36 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 14:29:36 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 14:32:57 mds1 kernel: INFO: task tgt_recov:5585 blocked for more than 120 seconds.
Jan 13 14:32:57 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 13 14:32:57 mds1 kernel: tgt_recov     D 0000000000000000     0  5585      2 0x00000080
Jan 13 14:32:57 mds1 kernel: ffff881006d47da0 0000000000000046 0000000000000000 ffff88085c456740
Jan 13 14:32:57 mds1 kernel: ffff881006d47d30 ffffffff81055f96 ffff881006d47d40 ffff881032616aa0
Jan 13 14:32:57 mds1 kernel: ffff881000fb65f8 ffff881006d47fd8 000000000000fb88 ffff881000fb65f8
Jan 13 14:32:57 mds1 kernel: Call Trace:
Jan 13 14:32:57 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 13 14:32:57 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 13 14:32:57 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 13 14:32:57 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 13 14:32:57 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 13 14:32:57 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 13 14:33:24 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 13 14:34:23 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 13 14:34:23 mds1 kernel: Lustre: emc-MDT0000: disconnecting 1 stale clients
Jan 13 14:34:23 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 5:00, of 7 clients 6 recovered and 1 was evicted.
Jan 13 14:48:18 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 13 14:48:18 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 14:48:18 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 14:48:18 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection to emc-OST0013 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 14:48:18 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 14:48:28 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection restored to emc-OST0014 (at 192.168.1.46@o2ib)
Jan 13 15:20:16 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800007160:0x4e:0x0], rc = 0 [1]
Jan 13 15:56:21 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 96ca9c86-3c3b-b3ac-00ee-5e96403ffb87 (at 192.168.1.55@o2ib) in 232 seconds. I think it's dead, and I am evicting it. exp ffff880825a6c800, cur 1389646581 expire 1389646431 last 1389646349
Jan 13 17:02:23 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 13 17:02:23 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 13 17:02:23 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 17:02:23 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 17:02:59 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 17:03:05 mds1 kernel: Lustre: 19122:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389650579/real 1389650579]  req@ffff881002674400 x1457144208739368/t0(0) o39->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1389650585 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:03:08 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 17:03:14 mds1 kernel: Lustre: 19196:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389650588/real 1389650588]  req@ffff881007197c00 x1457144208739568/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389650594 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:03:14 mds1 kernel: Lustre: server umount MGS complete
Jan 13 17:03:17 mds1 init: tty (/dev/tty1) main process (3871) killed by TERM signal
Jan 13 17:03:17 mds1 init: tty (/dev/tty2) main process (3873) killed by TERM signal
Jan 13 17:03:17 mds1 init: tty (/dev/tty3) main process (3875) killed by TERM signal
Jan 13 17:03:17 mds1 init: tty (/dev/tty4) main process (3877) killed by TERM signal
Jan 13 17:03:17 mds1 init: tty (/dev/tty5) main process (3879) killed by TERM signal
Jan 13 17:03:17 mds1 init: tty (/dev/tty6) main process (3881) killed by TERM signal
Jan 13 17:03:18 mds1 abrtd: Got signal 15, exiting
Jan 13 17:03:21 mds1 acpid: exiting
Jan 13 17:03:21 mds1 multipathd: mgt: event checker exit
Jan 13 17:03:21 mds1 multipathd: dm-0: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-0: devmap not registered, can't remove
Jan 13 17:03:21 mds1 kernel: sd 1:0:0:0: emc: Detached
Jan 13 17:03:21 mds1 kernel: sd 2:0:0:0: emc: Detached
Jan 13 17:03:21 mds1 multipathd: dm-0: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-0: devmap not registered, can't remove
Jan 13 17:03:21 mds1 multipathd: mdt1: event checker exit
Jan 13 17:03:21 mds1 kernel: sd 1:0:0:1: emc: Detached
Jan 13 17:03:21 mds1 kernel: sd 2:0:0:1: emc: Detached
Jan 13 17:03:21 mds1 multipathd: dm-1: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 13 17:03:21 mds1 multipathd: dm-1: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 13 17:03:21 mds1 multipathd: mdt2: event checker exit
Jan 13 17:03:21 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 17:03:21 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 13 17:03:21 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 13 17:03:21 mds1 multipathd: dm-2: remove map (uevent)
Jan 13 17:03:21 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 13 17:03:25 mds1 kernel: cib[3425]: segfault at 3d53e87de0 ip 0000003d51476205 sp 00007fff2c12fd80 error 4 in libc-2.12.so[3d51400000+18a000]
Jan 13 17:03:25 mds1 init: Disconnected from system bus
Jan 13 17:03:25 mds1 multipathd: --------shut down-------
Jan 13 17:03:25 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 13 17:03:25 mds1 auditd[3142]: The audit daemon is exiting.
Jan 13 17:03:25 mds1 kernel: type=1305 audit(1389650605.489:342): audit_pid=0 old=3142 auid=4294967295 ses=4294967295 res=1
Jan 13 17:03:25 mds1 kernel: type=1305 audit(1389650605.590:343): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Jan 13 17:03:25 mds1 kernel: Kernel logging (proc) stopped.
Jan 13 17:03:25 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3167" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 13 17:06:12 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 13 17:06:12 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3096" x-info="http://www.rsyslog.com"] start
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys cpuset
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys cpu
Jan 13 17:06:12 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 13 17:06:12 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 17:06:12 mds1 kernel: KERNEL supported cpus:
Jan 13 17:06:12 mds1 kernel:  Intel GenuineIntel
Jan 13 17:06:12 mds1 kernel:  AMD AuthenticAMD
Jan 13 17:06:12 mds1 kernel:  Centaur CentaurHauls
Jan 13 17:06:12 mds1 kernel: BIOS-provided physical RAM map:
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 13 17:06:12 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 13 17:06:12 mds1 kernel: DMI 2.6 present.
Jan 13 17:06:12 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 13 17:06:12 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 13 17:06:12 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 13 17:06:12 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 13 17:06:12 mds1 kernel: Using GB pages for direct mapping
Jan 13 17:06:12 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 13 17:06:12 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 13 17:06:12 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 13 17:06:12 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 13 17:06:12 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 13 17:06:12 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 13 17:06:12 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 13 17:06:12 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 17:06:12 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 17:06:12 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 17:06:12 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 13 17:06:12 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 13 17:06:12 mds1 kernel: Setting APIC routing to flat.
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 13 17:06:12 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 13 17:06:12 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 13 17:06:12 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 13 17:06:12 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 13 17:06:12 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 13 17:06:12 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 13 17:06:12 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 13 17:06:12 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 13 17:06:12 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 13 17:06:12 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 13 17:06:12 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 13 17:06:12 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 13 17:06:12 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 13 17:06:12 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 13 17:06:12 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 13 17:06:12 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 13 17:06:12 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 13 17:06:12 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 13 17:06:12 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 13 17:06:12 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 13 17:06:12 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 13 17:06:12 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 13 17:06:12 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 13 17:06:12 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 13 17:06:12 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 13 17:06:12 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 13 17:06:12 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 13 17:06:12 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 13 17:06:12 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 13 17:06:12 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 13 17:06:12 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 13 17:06:12 mds1 kernel: Zone PFN ranges:
Jan 13 17:06:12 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 13 17:06:12 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 13 17:06:12 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 13 17:06:12 mds1 kernel: Movable zone start PFN for each node
Jan 13 17:06:12 mds1 kernel: early_node_map[5] active PFN ranges
Jan 13 17:06:12 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 13 17:06:12 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 13 17:06:12 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 13 17:06:12 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 13 17:06:12 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 13 17:06:12 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 13 17:06:12 mds1 kernel: Setting APIC routing to flat.
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 13 17:06:12 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 13 17:06:12 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 13 17:06:12 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 13 17:06:12 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 13 17:06:12 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 13 17:06:12 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 13 17:06:12 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 13 17:06:12 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 13 17:06:12 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 13 17:06:12 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 13 17:06:12 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 13 17:06:12 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 13 17:06:12 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 13 17:06:12 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 13 17:06:12 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 13 17:06:12 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 13 17:06:12 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 13 17:06:12 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 13 17:06:12 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 13 17:06:12 mds1 kernel: Policy zone: Normal
Jan 13 17:06:12 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 13 17:06:12 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 13 17:06:12 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 13 17:06:12 mds1 kernel: Checking aperture...
Jan 13 17:06:12 mds1 kernel: No AGP bridge found
Jan 13 17:06:12 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 13 17:06:12 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 13 17:06:12 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 13 17:06:12 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 13 17:06:12 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 13 17:06:12 mds1 kernel: Hierarchical RCU implementation.
Jan 13 17:06:12 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 13 17:06:12 mds1 kernel: Extended CMOS year: 2000
Jan 13 17:06:12 mds1 kernel: Console: colour VGA+ 80x25
Jan 13 17:06:12 mds1 kernel: console [tty0] enabled
Jan 13 17:06:12 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 13 17:06:12 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 13 17:06:12 mds1 kernel: Fast TSC calibration using PIT
Jan 13 17:06:12 mds1 kernel: Detected 1995.326 MHz processor.
Jan 13 17:06:12 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.65 BogoMIPS (lpj=1995326)
Jan 13 17:06:12 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 13 17:06:12 mds1 kernel: Security Framework initialized
Jan 13 17:06:12 mds1 kernel: SELinux:  Initializing.
Jan 13 17:06:12 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 13 17:06:12 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 13 17:06:12 mds1 kernel: Mount-cache hash table entries: 256
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys ns
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys memory
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys devices
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys freezer
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys net_cls
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys blkio
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys perf_event
Jan 13 17:06:12 mds1 kernel: Initializing cgroup subsys net_prio
Jan 13 17:06:12 mds1 kernel: CPU: Physical Processor ID: 0
Jan 13 17:06:12 mds1 kernel: CPU: Processor Core ID: 0
Jan 13 17:06:12 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 13 17:06:12 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 13 17:06:12 mds1 kernel: using mwait in idle threads.
Jan 13 17:06:12 mds1 kernel: ACPI: Core revision 20090903
Jan 13 17:06:12 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 13 17:06:12 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 13 17:06:12 mds1 kernel: dmar: Host address width 46
Jan 13 17:06:12 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 13 17:06:12 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 17:06:12 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 13 17:06:12 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 13 17:06:12 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 13 17:06:12 mds1 kernel: dmar: ATSR flags: 0x0
Jan 13 17:06:12 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 13 17:06:12 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 13 17:06:12 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 13 17:06:12 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 13 17:06:12 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 13 17:06:12 mds1 kernel: Enabling x2apic
Jan 13 17:06:12 mds1 kernel: Enabled x2apic
Jan 13 17:06:12 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 13 17:06:12 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 13 17:06:12 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 13 17:06:12 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 13 17:06:12 mds1 kernel: PEBS disabled due to CPU errata.
Jan 13 17:06:12 mds1 kernel: ... version:                3
Jan 13 17:06:12 mds1 kernel: ... bit width:              48
Jan 13 17:06:12 mds1 kernel: ... generic registers:      8
Jan 13 17:06:12 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 13 17:06:12 mds1 kernel: ... max period:             000000007fffffff
Jan 13 17:06:12 mds1 kernel: ... fixed-purpose events:   3
Jan 13 17:06:12 mds1 kernel: ... event mask:             00000007000000ff
Jan 13 17:06:12 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 13 17:06:12 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 13 17:06:12 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 13 17:06:12 mds1 kernel: Brought up 12 CPUs
Jan 13 17:06:12 mds1 kernel: Total of 12 processors activated (47883.00 BogoMIPS).
Jan 13 17:06:12 mds1 kernel: devtmpfs: initialized
Jan 13 17:06:12 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 13 17:06:12 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 13 17:06:12 mds1 kernel: regulator: core version 0.5
Jan 13 17:06:12 mds1 kernel: NET: Registered protocol family 16
Jan 13 17:06:12 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 13 17:06:12 mds1 kernel: ACPI: bus type pci registered
Jan 13 17:06:12 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 13 17:06:12 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 13 17:06:12 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 13 17:06:12 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 13 17:06:12 mds1 kernel: bio: create slab <bio-0> at 0
Jan 13 17:06:12 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 13 17:06:12 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 13 17:06:12 mds1 kernel: ACPI: Interpreter enabled
Jan 13 17:06:12 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 13 17:06:12 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 13 17:06:12 mds1 kernel: ACPI: No dock devices found.
Jan 13 17:06:12 mds1 kernel: HEST: Table parsing has been initialized.
Jan 13 17:06:12 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 13 17:06:12 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 13 17:06:12 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 13 17:06:12 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 13 17:06:12 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 17:06:12 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 13 17:06:12 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 13 17:06:12 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 13 17:06:12 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 13 17:06:12 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 13 17:06:12 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 17:06:12 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 13 17:06:12 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 13 17:06:12 mds1 kernel: vgaarb: loaded
Jan 13 17:06:12 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 13 17:06:12 mds1 kernel: SCSI subsystem initialized
Jan 13 17:06:12 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 13 17:06:12 mds1 kernel: usbcore: registered new interface driver hub
Jan 13 17:06:12 mds1 kernel: usbcore: registered new device driver usb
Jan 13 17:06:12 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 13 17:06:12 mds1 kernel: NetLabel: Initializing
Jan 13 17:06:12 mds1 kernel: NetLabel:  domain hash size = 128
Jan 13 17:06:12 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 13 17:06:12 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 13 17:06:12 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 13 17:06:12 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 13 17:06:12 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 13 17:06:12 mds1 kernel: Switching to clocksource hpet
Jan 13 17:06:12 mds1 kernel: pnp: PnP ACPI init
Jan 13 17:06:12 mds1 kernel: ACPI: bus type pnp registered
Jan 13 17:06:12 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 13 17:06:12 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 13 17:06:12 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 13 17:06:12 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 17:06:12 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 17:06:12 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 13 17:06:12 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 17:06:12 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 13 17:06:12 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 17:06:12 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 17:06:12 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 13 17:06:12 mds1 kernel: NET: Registered protocol family 2
Jan 13 17:06:12 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 13 17:06:12 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 13 17:06:12 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 13 17:06:12 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 13 17:06:12 mds1 kernel: TCP reno registered
Jan 13 17:06:12 mds1 kernel: NET: Registered protocol family 1
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 17:06:12 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 13 17:06:12 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 13 17:06:12 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 13 17:06:12 mds1 kernel: dmar: Unsupported device scope
Jan 13 17:06:12 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 13 17:06:12 mds1 kernel: type=2000 audit(1389650751.057:1): initialized
Jan 13 17:06:12 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 13 17:06:12 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 13 17:06:12 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 13 17:06:12 mds1 kernel: msgmni has been set to 32768
Jan 13 17:06:12 mds1 kernel: alg: No test for stdrng (krng)
Jan 13 17:06:12 mds1 kernel: ksign: Installing public key data
Jan 13 17:06:12 mds1 kernel: Loading keyring
Jan 13 17:06:12 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 13 17:06:12 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 13 17:06:12 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 13 17:06:12 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 13 17:06:12 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 13 17:06:12 mds1 kernel: io scheduler noop registered
Jan 13 17:06:12 mds1 kernel: io scheduler anticipatory registered
Jan 13 17:06:12 mds1 kernel: io scheduler deadline registered (default)
Jan 13 17:06:12 mds1 kernel: io scheduler cfq registered
Jan 13 17:06:12 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 13 17:06:12 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 13 17:06:12 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 13 17:06:12 mds1 kernel: acpiphp: Slot [2] registered
Jan 13 17:06:12 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 13 17:06:12 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 13 17:06:12 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 13 17:06:12 mds1 kernel: ipmi message handler version 39.2
Jan 13 17:06:12 mds1 kernel: IPMI System Interface driver.
Jan 13 17:06:12 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 13 17:06:12 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 13 17:06:12 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 13 17:06:12 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 13 17:06:12 mds1 kernel: IPMI kcs interface initialized
Jan 13 17:06:12 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 13 17:06:12 mds1 kernel: ACPI: Power Button [PWRF]
Jan 13 17:06:12 mds1 kernel: ERST: Could not register with persistent store
Jan 13 17:06:12 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 13 17:06:12 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 13 17:06:12 mds1 kernel: Non-volatile memory driver v1.3
Jan 13 17:06:12 mds1 kernel: Linux agpgart interface v0.103
Jan 13 17:06:12 mds1 kernel: crash memory driver: version 1.1
Jan 13 17:06:12 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 13 17:06:12 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 17:06:12 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 17:06:12 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 13 17:06:12 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 13 17:06:12 mds1 kernel: brd: module loaded
Jan 13 17:06:12 mds1 kernel: loop: module loaded
Jan 13 17:06:12 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 13 17:06:12 mds1 kernel: Fixed MDIO Bus: probed
Jan 13 17:06:12 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 13 17:06:12 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 17:06:12 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 17:06:12 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 13 17:06:12 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 17:06:12 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 13 17:06:12 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 13 17:06:12 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 13 17:06:12 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 13 17:06:12 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 13 17:06:12 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 13 17:06:12 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 13 17:06:12 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 13 17:06:12 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 13 17:06:12 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 13 17:06:12 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 13 17:06:12 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 13 17:06:12 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 13 17:06:12 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 13 17:06:12 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 13 17:06:12 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 13 17:06:12 mds1 kernel: Switching to clocksource tsc
Jan 13 17:06:12 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 13 17:06:12 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 13 17:06:12 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 13 17:06:12 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 13 17:06:12 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 13 17:06:12 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 13 17:06:12 mds1 kernel: cpuidle: using governor ladder
Jan 13 17:06:12 mds1 kernel: cpuidle: using governor menu
Jan 13 17:06:12 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 13 17:06:12 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 13 17:06:12 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 13 17:06:12 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 13 17:06:12 mds1 kernel: TCP cubic registered
Jan 13 17:06:12 mds1 kernel: Initializing XFRM netlink socket
Jan 13 17:06:12 mds1 kernel: NET: Registered protocol family 17
Jan 13 17:06:12 mds1 kernel: registered taskstats version 1
Jan 13 17:06:12 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-13 22:05:53 UTC (1389650753)
Jan 13 17:06:12 mds1 kernel: Initalizing network drop monitor service
Jan 13 17:06:12 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 13 17:06:12 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 13 17:06:12 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 13 17:06:12 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 13 17:06:12 mds1 kernel: dracut: dracut-004-303.el6
Jan 13 17:06:12 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 13 17:06:12 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 13 17:06:12 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 13 17:06:12 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 13 17:06:12 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 17:06:12 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 17:06:12 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 13 17:06:12 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 13 17:06:12 mds1 kernel: udev: starting version 147
Jan 13 17:06:12 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 13 17:06:12 mds1 kernel: dracut: Starting plymouth daemon
Jan 13 17:06:12 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 13 17:06:12 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 13 17:06:12 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 13 17:06:12 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 13 17:06:12 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 13 17:06:12 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 13 17:06:12 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 13 17:06:12 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 13 17:06:12 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 13 17:06:12 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 13 17:06:12 mds1 kernel: megasas: FW now in Ready state
Jan 13 17:06:12 mds1 kernel: megasas:IOC Init cmd success
Jan 13 17:06:12 mds1 kernel: megasas: INIT adapter done
Jan 13 17:06:12 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 13 17:06:12 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 17:06:12 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 13 17:06:12 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 13 17:06:12 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 13 17:06:12 mds1 kernel: scsi1 : qla2xxx
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 13 17:06:12 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 13 17:06:12 mds1 kernel: scsi2 : qla2xxx
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 13 17:06:12 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 13 17:06:12 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 13 17:06:12 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 13 17:06:12 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 13 17:06:12 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 13 17:06:12 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 13 17:06:12 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 13 17:06:12 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 13 17:06:12 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2da6, mb[3]=e100, cmd=6a ****.
Jan 13 17:06:12 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2d46, mb[3]=9100, cmd=6a ****.
Jan 13 17:06:12 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 13 17:06:12 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 13 17:06:12 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 13 17:06:12 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 13 17:06:12 mds1 kernel: scsi3 : ahci
Jan 13 17:06:12 mds1 kernel: scsi4 : ahci
Jan 13 17:06:12 mds1 kernel: scsi5 : ahci
Jan 13 17:06:12 mds1 kernel: scsi6 : ahci
Jan 13 17:06:12 mds1 kernel: scsi7 : ahci
Jan 13 17:06:12 mds1 kernel: scsi8 : ahci
Jan 13 17:06:12 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 13 17:06:12 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 13 17:06:12 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 13 17:06:12 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 13 17:06:12 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 13 17:06:12 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 13 17:06:12 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 13 17:06:12 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 13 17:06:12 mds1 kernel: ata1.00: configured for UDMA/100
Jan 13 17:06:12 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 13 17:06:12 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 13 17:06:12 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:12 mds1 kernel: sda:
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sdc:
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:12 mds1 kernel: sdb: unknown partition table
Jan 13 17:06:12 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 13 17:06:12 mds1 kernel: sdd:
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 17:06:13 mds1 kernel: unknown partition table
Jan 13 17:06:13 mds1 kernel: unknown partition table
Jan 13 17:06:13 mds1 kernel: sdg:
Jan 13 17:06:13 mds1 kernel: sdf:
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sde: unknown partition table
Jan 13 17:06:13 mds1 kernel: unknown partition table
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: unknown partition table
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sda1 sda2 sda3
Jan 13 17:06:13 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 13 17:06:13 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 13 17:06:13 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 13 17:06:13 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 13 17:06:13 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 13 17:06:13 mds1 kernel: SELinux:  Disabled at runtime.
Jan 13 17:06:13 mds1 kernel: type=1404 audit(1389650756.294:2): selinux=0 auid=4294967295 ses=4294967295
Jan 13 17:06:13 mds1 kernel: dracut: 
Jan 13 17:06:13 mds1 kernel: dracut: Switching root
Jan 13 17:06:13 mds1 kernel: readahead: starting
Jan 13 17:06:13 mds1 kernel: udev: starting version 147
Jan 13 17:06:13 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 13 17:06:13 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 13 17:06:13 mds1 kernel: PTP clock support registered
Jan 13 17:06:13 mds1 kernel: dca service started, version 1.12.1
Jan 13 17:06:13 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 13 17:06:13 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 13 17:06:13 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 13 17:06:13 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 13 17:06:13 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 13 17:06:13 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 13 17:06:13 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 13 17:06:13 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 13 17:06:13 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 13 17:06:13 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 13 17:06:13 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 13 17:06:13 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 13 17:06:13 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 13 17:06:13 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 13 17:06:13 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 13 17:06:13 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 13 17:06:13 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 13 17:06:13 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 13 17:06:13 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 13 17:06:13 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 13 17:06:13 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 13 17:06:13 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 13 17:06:13 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 13 17:06:13 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 13 17:06:13 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 17:06:13 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 17:06:13 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 13 17:06:13 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 17:06:13 mds1 kernel: emc: device handler registered
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 13 17:06:13 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 13 17:06:13 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 13 17:06:13 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 13 17:06:13 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 13 17:06:13 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 13 17:06:13 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 13 17:06:13 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 13 17:06:13 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102888b000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 13 17:06:13 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102888b000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 13 17:06:13 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 13 17:06:13 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 13 17:06:13 mds1 kernel: NET: Registered protocol family 10
Jan 13 17:06:13 mds1 kernel: lo: Disabled Privacy Extensions
Jan 13 17:06:13 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 17:06:13 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 13 17:06:13 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 13 17:06:13 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 17:06:13 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 17:06:13 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 13 17:06:13 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 13 17:06:13 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 13 17:06:13 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 13 17:06:13 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 13 17:06:13 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 13 17:06:13 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 13 17:06:13 mds1 rpc.statd[3191]: Version 1.2.3 starting
Jan 13 17:06:13 mds1 sm-notify[3192]: Version 1.2.3 starting
Jan 13 17:06:13 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 13 17:06:13 mds1 kernel: RPC: Registered udp transport module.
Jan 13 17:06:13 mds1 kernel: RPC: Registered tcp transport module.
Jan 13 17:06:13 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 13 17:06:13 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 13 17:06:13 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 13 17:06:13 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 13 17:06:13 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 13 17:06:17 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 13 17:06:17 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 13 17:06:21 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 13 17:06:22 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 13 17:06:22 mds1 cib[3334]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:22 mds1 attrd[3337]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:22 mds1 pengine[3338]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:22 mds1 crmd[3339]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:22 mds1 stonith-ng[3335]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:22 mds1 lrmd[3336]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:06:23 mds1 logger[3348]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 13 17:06:24 mds1 kdump: kexec: loaded kdump kernel
Jan 13 17:06:24 mds1 kdump: started up
Jan 13 17:06:26 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 13 17:06:26 mds1 kernel: Slow work thread pool: Starting up
Jan 13 17:06:26 mds1 kernel: Slow work thread pool: Ready
Jan 13 17:06:26 mds1 kernel: FS-Cache: Loaded
Jan 13 17:06:26 mds1 kernel: Registering the id_resolver key type
Jan 13 17:06:26 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching
Jan 13 17:06:26 mds1 acpid: starting up
Jan 13 17:06:26 mds1 acpid: 1 rule loaded
Jan 13 17:06:26 mds1 acpid: waiting for events: event logging is off
Jan 13 17:06:27 mds1 acpid: client connected from 3567[68:68]
Jan 13 17:06:27 mds1 acpid: 1 client rule loaded
Jan 13 17:06:27 mds1 kernel: ipmi device interface
Jan 13 17:06:28 mds1 automount[3606]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 13 17:06:29 mds1 abrtd: Init complete, entering main loop
Jan 13 17:06:29 mds1 cimserver[3757]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 13 17:06:33 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 13 17:06:35 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 13 17:14:33 mds1 cibadmin[4406]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 17:14:33 mds1 cibadmin[4407]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 17:14:33 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 17:14:35 mds1 cibadmin[4570]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 17:14:35 mds1 cibadmin[4571]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 17:14:36 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 17:14:36 mds1 cibadmin[4656]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 17:14:36 mds1 cibadmin[4657]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 17:14:36 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 17:14:36 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 17:14:36 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 17:14:36 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 17:14:41 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651276/real 0]  req@ffff880fe7557800 x1457154457338040/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651281 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:14:48 mds1 cibadmin[4807]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 13 17:15:01 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 13 17:15:01 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 13 17:15:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651301/real 0]  req@ffff880fe772a400 x1457154457338260/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651306 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:15:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 13 17:15:12 mds1 crmd[4838]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 13 17:15:26 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 13 17:15:36 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651326/real 0]  req@ffff881011711c00 x1457154457338364/t0(0) o8->emc-OST0016-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651336 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:15:36 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:16:01 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651351/real 0]  req@ffff880fe79e9800 x1457154457338480/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651361 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:16:01 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:16:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651376/real 0]  req@ffff880fe9d0f400 x1457154457338588/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651391 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:16:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:16:56 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651401/real 0]  req@ffff880fe7765000 x1457154457338696/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651416 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:16:56 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:17:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651426/real 0]  req@ffff880fe764c000 x1457154457338804/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651446 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:17:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:17:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651451/real 0]  req@ffff880fe764a000 x1457154457338912/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651471 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:17:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 17:18:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389651526/real 1389651526]  req@ffff880ffc046400 x1457154457339052/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651551 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:18:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 17:20:01 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389651601/real 1389651601]  req@ffff880fe774c000 x1457154457339324/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:20:01 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 13 17:22:21 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389651701/real 0]  req@ffff880fe7677800 x1457154457339656/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389651741 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:22:21 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 83 previous similar messages
Jan 13 17:26:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389651951/real 1389651951]  req@ffff88102fc9e000 x1457154457340360/t0(0) o8->emc-OST000d-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389652006 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:26:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 143 previous similar messages
Jan 13 17:35:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389652526/real 1389652526]  req@ffff880fea751800 x1457154457341688/t0(0) o8->emc-OST000c-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389652581 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:35:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 251 previous similar messages
Jan 13 17:45:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389653126/real 1389653126]  req@ffff881011108800 x1457154457343160/t0(0) o8->emc-OST0014-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389653181 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:45:26 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 295 previous similar messages
Jan 13 17:55:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389653676/real 1389653676]  req@ffff88082302f000 x1457154457344540/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389653731 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 17:55:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 291 previous similar messages
Jan 13 18:05:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389654351/real 1389654351]  req@ffff8808230a6400 x1457154457346072/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389654406 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:05:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 299 previous similar messages
Jan 13 18:15:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389654951/real 1389654951]  req@ffff880826935c00 x1457154457347544/t0(0) o8->emc-OST0002-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389655006 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:15:51 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 295 previous similar messages
Jan 13 18:26:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389655551/real 1389655551]  req@ffff880826921000 x1457154457349036/t0(0) o8->emc-OST0016-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389655606 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:26:46 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 291 previous similar messages
Jan 13 18:37:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389656226/real 1389656226]  req@ffff8808269f6800 x1457154457350568/t0(0) o8->emc-OST000c-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389656281 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:37:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 299 previous similar messages
Jan 13 18:47:11 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389656776/real 1389656776]  req@ffff880ffc0f2c00 x1457154457351980/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389656831 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:47:11 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 287 previous similar messages
Jan 13 18:48:46 mds1 kernel: LustreError: 11-0: emc-OST000c-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -19.
Jan 13 18:49:36 mds1 kernel: LustreError: 11-0: emc-OST0008-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:49:36 mds1 kernel: LustreError: Skipped 7 previous similar messages
Jan 13 18:49:38 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:49:38 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 13 18:49:41 mds1 kernel: LustreError: 11-0: emc-OST000a-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:49:41 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 13 18:49:43 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 13 18:50:06 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:50:06 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 18:50:31 mds1 kernel: LustreError: 11-0: emc-OST000a-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:50:31 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 18:50:56 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:50:56 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 18:51:21 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:51:21 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 13 18:52:11 mds1 kernel: LustreError: 11-0: emc-OST000b-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 13 18:52:11 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 13 18:55:33 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection to emc-OST000c (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 18:55:33 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 18:55:33 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection to emc-OST000a (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 18:55:33 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 13 18:55:33 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 18:55:33 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 13 18:55:35 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 18:55:35 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 13 18:57:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389657435/real 0]  req@ffff8810305e3c00 x1457154457359748/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389657451 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 18:57:31 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 179 previous similar messages
Jan 13 18:59:45 mds1 kernel: Lustre: MGS: haven't heard from client 0039c62c-1f42-24dd-a0d2-d65b8b438e8a (at 192.168.1.44@o2ib) in 262 seconds. I think it's dead, and I am evicting it. exp ffff8810306cf400, cur 1389657585 expire 1389657435 last 1389657323
Jan 13 18:59:45 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST000f_UUID (at 192.168.1.44@o2ib) in 262 seconds. I think it's dead, and I am evicting it. exp ffff880831f91800, cur 1389657585 expire 1389657435 last 1389657323
Jan 13 18:59:45 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 13 19:08:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389658035/real 1389658035]  req@ffff880826ed3400 x1457154457361380/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389658086 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 19:08:06 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 335 previous similar messages
Jan 13 19:08:13 mds1 kernel: Lustre: MGS: Client 6d4a8c65-0c77-24d8-906a-df7b153fb09e (at 192.168.1.44@o2ib) reconnecting
Jan 13 19:08:20 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.44@o2ib)
Jan 13 19:08:20 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.44@o2ib)
Jan 13 19:08:20 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 13 19:21:10 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 13 19:21:10 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection to emc-OST0008 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 19:21:10 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 19:21:10 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 13 19:21:20 mds1 kernel: Lustre: MGS: Client 42ac5928-4eb2-0f91-15b4-8eb876fcaf5e (at 192.168.1.46@o2ib) reconnecting
Jan 13 19:21:30 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 13 19:21:30 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 13 19:21:36 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection restored to emc-OST000a (at 192.168.1.44@o2ib)
Jan 13 19:21:36 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 13 20:14:43 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 20:14:44 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 20:14:44 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 20:14:44 mds1 kernel: Lustre: emc-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 13 20:14:50 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 2:30, or until 5 clients reconnect
Jan 13 20:15:16 mds1 kernel: Lustre: 3319:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389662109/real 1389662109]  req@ffff880826c6a800 x1457154457444728/t0(0) o400->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1389662116 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 20:15:16 mds1 kernel: Lustre: 3319:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 13 20:15:16 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 20:15:16 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 13 20:15:41 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:51, of 5 clients 5 recovered and 0 were evicted.
Jan 13 20:15:41 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 13 20:15:41 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 13 20:31:29 mds1 kernel: Lustre: MGS: haven't heard from client ba474423-8ea6-3ec8-113f-aa8f1843676b (at 192.168.1.57@o2ib) in 240 seconds. I think it's dead, and I am evicting it. exp ffff880826b87000, cur 1389663089 expire 1389662939 last 1389662849
Jan 13 20:31:29 mds1 kernel: Lustre: Skipped 17 previous similar messages
Jan 13 20:31:31 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 45525470-037b-694d-3881-8085c45ced73 (at 192.168.1.57@o2ib) in 242 seconds. I think it's dead, and I am evicting it. exp ffff88082020fc00, cur 1389663091 expire 1389662941 last 1389662849
Jan 13 20:31:31 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 20:32:45 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client a3768068-f6d2-64c4-3fa5-df58777ce7e6 (at 192.168.1.54@o2ib) in 165 seconds. I think it's dead, and I am evicting it. exp ffff880fe74c4000, cur 1389663165 expire 1389663015 last 1389663000
Jan 13 20:33:59 mds1 kernel: Lustre: MGS: haven't heard from client 232dbb44-66e6-2ee6-3686-685531be1b27 (at 192.168.1.56@o2ib) in 247 seconds. I think it's dead, and I am evicting it. exp ffff88082e33e000, cur 1389663239 expire 1389663089 last 1389662992
Jan 13 20:33:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 20:34:01 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 68c9d57f-4e43-c1cc-0024-940c69a956b7 (at 192.168.1.56@o2ib) in 249 seconds. I think it's dead, and I am evicting it. exp ffff88082020e000, cur 1389663241 expire 1389663091 last 1389662992
Jan 13 20:34:01 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 13 20:43:31 mds1 kernel: LustreError: 11-0: emc-OST0013-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 13 20:43:31 mds1 kernel: Lustre: emc-OST000c-osc-MDT0001: Connection to emc-OST000c (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 20:43:31 mds1 kernel: LustreError: Skipped 22 previous similar messages
Jan 13 20:43:38 mds1 kernel: Lustre: 3318:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389663811/real 1389663811]  req@ffff880fe66dd400 x1457154457512592/t0(0) o13->emc-OST0001-osc-MDT0001@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389663818 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 13 20:43:38 mds1 kernel: Lustre: 3318:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 13 20:43:38 mds1 kernel: Lustre: emc-OST0006-osc-MDT0001: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 20:43:38 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 13 20:43:42 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 20:43:42 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 13 20:43:43 mds1 kernel: Lustre: emc-OST0017-osc-MDT0001: Connection to emc-OST0017 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 13 20:43:43 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 13 20:43:47 mds1 cibadmin[23909]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:43:47 mds1 cibadmin[23910]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:43:47 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 13 20:43:48 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 13 20:43:48 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389663822/real 1389663822]  req@ffff881011108000 x1457154457512820/t0(0) o8->emc-OST0012-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389663828 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 20:43:48 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 60 previous similar messages
Jan 13 20:43:49 mds1 cibadmin[23987]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:43:49 mds1 cibadmin[23988]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:43:49 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 13 20:43:49 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_disconnect failed with -107.
Jan 13 20:43:49 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 13 20:43:51 mds1 cibadmin[24065]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:43:51 mds1 cibadmin[24066]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:43:57 mds1 kernel: Lustre: server umount MGS complete
Jan 13 20:44:07 mds1 cibadmin[24171]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:44:07 mds1 cibadmin[24172]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:44:07 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 20:44:09 mds1 cibadmin[24329]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:44:09 mds1 cibadmin[24330]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:44:10 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 20:44:11 mds1 cibadmin[24457]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 13 20:44:11 mds1 cibadmin[24458]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 13 20:44:11 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 13 20:44:11 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 20:44:11 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 20:44:11 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 20:44:11 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 20:44:12 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 13 20:44:12 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 13 20:44:12 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 13 20:44:16 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389663851/real 1389663851]  req@ffff880fe710e000 x1457154457513048/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389663856 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 13 20:44:16 mds1 kernel: Lustre: 3307:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jan 13 20:44:38 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 13 20:44:38 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 13 20:44:38 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 13 20:44:38 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 13 20:44:38 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 13 20:44:39 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 13 20:44:49 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection restored to emc-OST0005 (at 192.168.1.44@o2ib)
Jan 13 20:44:49 mds1 kernel: Lustre: Skipped 19 previous similar messages
Jan 14 07:55:12 mds1 auditd[3071]: Audit daemon rotating log files
Jan 14 10:27:43 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008101:0x4911:0x0], rc = 0 [1]
Jan 14 10:27:49 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008101:0x4911:0x0], rc = 0 [1]
Jan 14 10:27:55 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008101:0x4911:0x0], rc = 0 [1]
Jan 14 10:28:01 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008101:0x4911:0x0], rc = 0 [1]
Jan 14 14:17:38 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008102:0x1d7:0x0], rc = 0 [1]
Jan 14 14:17:44 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008102:0x1d7:0x0], rc = 0 [1]
Jan 14 14:22:08 mds1 kernel: Lustre: MGS: haven't heard from client 527c14be-3733-bfc4-a146-97d04ae5370c (at 192.168.1.53@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880fd2db5000, cur 1389727328 expire 1389727178 last 1389727099
Jan 14 14:22:08 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 14:22:15 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client a97ebd02-b2ca-9378-90ea-515f9f70abec (at 192.168.1.53@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff881004213400, cur 1389727335 expire 1389727185 last 1389727106
Jan 14 14:22:15 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 14:28:38 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x800008103:0xb8:0x0], rc = 0 [1]
Jan 14 14:52:08 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 076afa40-9597-cd2b-d9d7-ab5585c8ebd3 (at 192.168.1.51@o2ib) in 243 seconds. I think it's dead, and I am evicting it. exp ffff88100a2e4000, cur 1389729128 expire 1389728978 last 1389728885
Jan 14 14:52:10 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 076afa40-9597-cd2b-d9d7-ab5585c8ebd3 (at 192.168.1.51@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd357f400, cur 1389729130 expire 1389728980 last 1389728903
Jan 14 14:52:10 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 15:08:25 mds1 kernel: Lustre: MGS: haven't heard from client 2f1e4b5b-7bee-1269-8ff4-fcbde1b9cde9 (at 192.168.1.55@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff880fe010c400, cur 1389730105 expire 1389729955 last 1389729875
Jan 14 15:08:29 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 00103b9b-8813-6d87-e521-53825b1fa7e8 (at 192.168.1.55@o2ib) in 234 seconds. I think it's dead, and I am evicting it. exp ffff8808219a7000, cur 1389730109 expire 1389729959 last 1389729875
Jan 14 15:17:06 mds1 kernel: Lustre: 3315:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389730619/real 0]  req@ffff880fe66dfc00 x1457154460487908/t0(0) o13->emc-OST0001-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389Jan 14 15:21:37 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 14 15:21:37 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3161" x-info="http://www.rsyslog.com"] start
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys cpuset
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys cpu
Jan 14 15:21:37 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 14 15:21:37 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 15:21:37 mds1 kernel: KERNEL supported cpus:
Jan 14 15:21:37 mds1 kernel:  Intel GenuineIntel
Jan 14 15:21:37 mds1 kernel:  AMD AuthenticAMD
Jan 14 15:21:37 mds1 kernel:  Centaur CentaurHauls
Jan 14 15:21:37 mds1 kernel: BIOS-provided physical RAM map:
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 14 15:21:37 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 14 15:21:37 mds1 kernel: DMI 2.6 present.
Jan 14 15:21:37 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 14 15:21:37 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 14 15:21:37 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 14 15:21:37 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 14 15:21:37 mds1 kernel: Using GB pages for direct mapping
Jan 14 15:21:37 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 14 15:21:37 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 14 15:21:37 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 14 15:21:37 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 14 15:21:37 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 14 15:21:37 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 14 15:21:37 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:21:37 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:21:37 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:21:37 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:21:37 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:21:37 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 14 15:21:37 mds1 kernel: Setting APIC routing to flat.
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 14 15:21:37 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 14 15:21:37 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 14 15:21:37 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 14 15:21:37 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 14 15:21:37 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 14 15:21:37 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 14 15:21:37 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 14 15:21:37 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 14 15:21:37 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 14 15:21:37 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 14 15:21:37 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 14 15:21:37 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 14 15:21:37 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 14 15:21:37 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 14 15:21:37 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 14 15:21:37 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 14 15:21:37 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 14 15:21:37 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 14 15:21:37 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 14 15:21:37 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 14 15:21:37 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 14 15:21:37 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 14 15:21:37 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 14 15:21:37 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 14 15:21:37 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 14 15:21:37 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 14 15:21:37 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 14 15:21:37 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 14 15:21:37 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 14 15:21:37 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 14 15:21:37 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 14 15:21:37 mds1 kernel: Zone PFN ranges:
Jan 14 15:21:37 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 14 15:21:37 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 14 15:21:37 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 14 15:21:37 mds1 kernel: Movable zone start PFN for each node
Jan 14 15:21:37 mds1 kernel: early_node_map[5] active PFN ranges
Jan 14 15:21:37 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 14 15:21:37 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 14 15:21:37 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 14 15:21:37 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 14 15:21:37 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 14 15:21:37 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 14 15:21:37 mds1 kernel: Setting APIC routing to flat.
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 14 15:21:37 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 14 15:21:37 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 14 15:21:37 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 14 15:21:37 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 14 15:21:37 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 14 15:21:37 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 14 15:21:37 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 14 15:21:37 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 14 15:21:37 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 14 15:21:37 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 14 15:21:37 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 14 15:21:37 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 14 15:21:37 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 14 15:21:37 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 14 15:21:37 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 14 15:21:37 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 14 15:21:37 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 14 15:21:37 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 14 15:21:37 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 14 15:21:37 mds1 kernel: Policy zone: Normal
Jan 14 15:21:37 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 15:21:37 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 14 15:21:37 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 14 15:21:37 mds1 kernel: Checking aperture...
Jan 14 15:21:37 mds1 kernel: No AGP bridge found
Jan 14 15:21:37 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 14 15:21:37 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 14 15:21:37 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 14 15:21:37 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 14 15:21:37 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 14 15:21:37 mds1 kernel: Hierarchical RCU implementation.
Jan 14 15:21:37 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 14 15:21:37 mds1 kernel: Extended CMOS year: 2000
Jan 14 15:21:37 mds1 kernel: Console: colour VGA+ 80x25
Jan 14 15:21:37 mds1 kernel: console [tty0] enabled
Jan 14 15:21:37 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 14 15:21:37 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 14 15:21:37 mds1 kernel: Fast TSC calibration using PIT
Jan 14 15:21:37 mds1 kernel: Detected 1995.201 MHz processor.
Jan 14 15:21:37 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.40 BogoMIPS (lpj=1995201)
Jan 14 15:21:37 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 14 15:21:37 mds1 kernel: Security Framework initialized
Jan 14 15:21:37 mds1 kernel: SELinux:  Initializing.
Jan 14 15:21:37 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 14 15:21:37 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 14 15:21:37 mds1 kernel: Mount-cache hash table entries: 256
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys ns
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys memory
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys devices
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys freezer
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys net_cls
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys blkio
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys perf_event
Jan 14 15:21:37 mds1 kernel: Initializing cgroup subsys net_prio
Jan 14 15:21:37 mds1 kernel: CPU: Physical Processor ID: 0
Jan 14 15:21:37 mds1 kernel: CPU: Processor Core ID: 0
Jan 14 15:21:37 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 14 15:21:37 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 14 15:21:37 mds1 kernel: using mwait in idle threads.
Jan 14 15:21:37 mds1 kernel: ACPI: Core revision 20090903
Jan 14 15:21:37 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 14 15:21:37 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 14 15:21:37 mds1 kernel: dmar: Host address width 46
Jan 14 15:21:37 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 14 15:21:37 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 15:21:37 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 14 15:21:37 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 15:21:37 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 14 15:21:37 mds1 kernel: dmar: ATSR flags: 0x0
Jan 14 15:21:37 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 14 15:21:37 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 14 15:21:37 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 14 15:21:37 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 14 15:21:37 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 14 15:21:37 mds1 kernel: Enabling x2apic
Jan 14 15:21:37 mds1 kernel: Enabled x2apic
Jan 14 15:21:37 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 14 15:21:37 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 14 15:21:37 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 14 15:21:37 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 14 15:21:37 mds1 kernel: PEBS disabled due to CPU errata.
Jan 14 15:21:37 mds1 kernel: ... version:                3
Jan 14 15:21:37 mds1 kernel: ... bit width:              48
Jan 14 15:21:37 mds1 kernel: ... generic registers:      8
Jan 14 15:21:37 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 14 15:21:37 mds1 kernel: ... max period:             000000007fffffff
Jan 14 15:21:37 mds1 kernel: ... fixed-purpose events:   3
Jan 14 15:21:37 mds1 kernel: ... event mask:             00000007000000ff
Jan 14 15:21:37 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 14 15:21:37 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 14 15:21:37 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 14 15:21:37 mds1 kernel: Brought up 12 CPUs
Jan 14 15:21:37 mds1 kernel: Total of 12 processors activated (47881.48 BogoMIPS).
Jan 14 15:21:37 mds1 kernel: devtmpfs: initialized
Jan 14 15:21:37 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 14 15:21:37 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 14 15:21:37 mds1 kernel: regulator: core version 0.5
Jan 14 15:21:37 mds1 kernel: NET: Registered protocol family 16
Jan 14 15:21:37 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 14 15:21:37 mds1 kernel: ACPI: bus type pci registered
Jan 14 15:21:37 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 14 15:21:37 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 14 15:21:37 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 14 15:21:37 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 14 15:21:37 mds1 kernel: bio: create slab <bio-0> at 0
Jan 14 15:21:37 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 14 15:21:37 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 14 15:21:37 mds1 kernel: ACPI: Interpreter enabled
Jan 14 15:21:37 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 14 15:21:37 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 14 15:21:37 mds1 kernel: ACPI: No dock devices found.
Jan 14 15:21:37 mds1 kernel: HEST: Table parsing has been initialized.
Jan 14 15:21:37 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 14 15:21:37 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 14 15:21:37 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 14 15:21:37 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 14 15:21:37 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:21:37 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 14 15:21:37 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 14 15:21:37 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 14 15:21:37 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 14 15:21:37 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 14 15:21:37 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:21:37 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:21:37 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 14 15:21:37 mds1 kernel: vgaarb: loaded
Jan 14 15:21:37 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 14 15:21:37 mds1 kernel: SCSI subsystem initialized
Jan 14 15:21:37 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 14 15:21:37 mds1 kernel: usbcore: registered new interface driver hub
Jan 14 15:21:37 mds1 kernel: usbcore: registered new device driver usb
Jan 14 15:21:37 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 14 15:21:37 mds1 kernel: NetLabel: Initializing
Jan 14 15:21:37 mds1 kernel: NetLabel:  domain hash size = 128
Jan 14 15:21:37 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 14 15:21:37 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 14 15:21:37 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 14 15:21:37 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 14 15:21:37 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 14 15:21:37 mds1 kernel: Switching to clocksource hpet
Jan 14 15:21:37 mds1 kernel: pnp: PnP ACPI init
Jan 14 15:21:37 mds1 kernel: ACPI: bus type pnp registered
Jan 14 15:21:37 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 14 15:21:37 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 14 15:21:37 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 14 15:21:37 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:21:37 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:21:37 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:21:37 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:21:37 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 14 15:21:37 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:21:37 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:21:37 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:21:37 mds1 kernel: NET: Registered protocol family 2
Jan 14 15:21:37 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 14 15:21:37 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 14 15:21:37 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 14 15:21:37 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 14 15:21:37 mds1 kernel: TCP reno registered
Jan 14 15:21:37 mds1 kernel: NET: Registered protocol family 1
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 15:21:37 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 14 15:21:37 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 14 15:21:37 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 14 15:21:37 mds1 kernel: dmar: Unsupported device scope
Jan 14 15:21:37 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 14 15:21:37 mds1 kernel: type=2000 audit(1389730874.055:1): initialized
Jan 14 15:21:37 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 14 15:21:37 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 14 15:21:37 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 14 15:21:37 mds1 kernel: msgmni has been set to 32768
Jan 14 15:21:37 mds1 kernel: alg: No test for stdrng (krng)
Jan 14 15:21:37 mds1 kernel: ksign: Installing public key data
Jan 14 15:21:37 mds1 kernel: Loading keyring
Jan 14 15:21:37 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 14 15:21:37 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 14 15:21:37 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 14 15:21:37 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 14 15:21:37 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 14 15:21:37 mds1 kernel: io scheduler noop registered
Jan 14 15:21:37 mds1 kernel: io scheduler anticipatory registered
Jan 14 15:21:37 mds1 kernel: io scheduler deadline registered (default)
Jan 14 15:21:37 mds1 kernel: io scheduler cfq registered
Jan 14 15:21:37 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 14 15:21:37 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 14 15:21:37 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 14 15:21:37 mds1 kernel: acpiphp: Slot [2] registered
Jan 14 15:21:37 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 14 15:21:37 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 14 15:21:37 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 14 15:21:37 mds1 kernel: ipmi message handler version 39.2
Jan 14 15:21:37 mds1 kernel: IPMI System Interface driver.
Jan 14 15:21:37 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 14 15:21:37 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 14 15:21:37 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 14 15:21:37 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 14 15:21:37 mds1 kernel: IPMI kcs interface initialized
Jan 14 15:21:37 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 14 15:21:37 mds1 kernel: ACPI: Power Button [PWRF]
Jan 14 15:21:37 mds1 kernel: ERST: Could not register with persistent store
Jan 14 15:21:37 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 14 15:21:37 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 14 15:21:37 mds1 kernel: Non-volatile memory driver v1.3
Jan 14 15:21:37 mds1 kernel: Linux agpgart interface v0.103
Jan 14 15:21:37 mds1 kernel: crash memory driver: version 1.1
Jan 14 15:21:37 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 14 15:21:37 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 15:21:37 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 15:21:37 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 15:21:37 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 15:21:37 mds1 kernel: brd: module loaded
Jan 14 15:21:37 mds1 kernel: loop: module loaded
Jan 14 15:21:37 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 14 15:21:37 mds1 kernel: Fixed MDIO Bus: probed
Jan 14 15:21:37 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 14 15:21:37 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 15:21:37 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 15:21:37 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 14 15:21:37 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 15:21:37 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 14 15:21:37 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 14 15:21:37 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 14 15:21:37 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 14 15:21:37 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 15:21:37 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 15:21:37 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 14 15:21:37 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 15:21:37 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 14 15:21:37 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 14 15:21:37 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 14 15:21:37 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 14 15:21:37 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 14 15:21:37 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 14 15:21:37 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 14 15:21:37 mds1 kernel: Switching to clocksource tsc
Jan 14 15:21:37 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 14 15:21:37 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 14 15:21:37 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 14 15:21:37 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 14 15:21:37 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 14 15:21:37 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 14 15:21:37 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 14 15:21:37 mds1 kernel: cpuidle: using governor ladder
Jan 14 15:21:37 mds1 kernel: cpuidle: using governor menu
Jan 14 15:21:37 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 14 15:21:37 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 14 15:21:37 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 14 15:21:37 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 14 15:21:37 mds1 kernel: TCP cubic registered
Jan 14 15:21:37 mds1 kernel: Initializing XFRM netlink socket
Jan 14 15:21:37 mds1 kernel: NET: Registered protocol family 17
Jan 14 15:21:37 mds1 kernel: registered taskstats version 1
Jan 14 15:21:37 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-14 20:21:16 UTC (1389730876)
Jan 14 15:21:37 mds1 kernel: Initalizing network drop monitor service
Jan 14 15:21:37 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 14 15:21:37 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 14 15:21:37 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 14 15:21:37 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 14 15:21:37 mds1 kernel: dracut: dracut-004-303.el6
Jan 14 15:21:37 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 14 15:21:37 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 14 15:21:37 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 14 15:21:37 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 14 15:21:37 mds1 kernel: udev: starting version 147
Jan 14 15:21:37 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 15:21:37 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 15:21:37 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 14 15:21:37 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 14 15:21:37 mds1 kernel: dracut: Starting plymouth daemon
Jan 14 15:21:37 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 14 15:21:37 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 14 15:21:37 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 14 15:21:37 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 14 15:21:37 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 14 15:21:37 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 14 15:21:37 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 14 15:21:37 mds1 kernel: megasas: FW now in Ready state
Jan 14 15:21:37 mds1 kernel: megasas:IOC Init cmd success
Jan 14 15:21:37 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 15:21:37 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 15:21:37 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 14 15:21:37 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 14 15:21:37 mds1 kernel: megasas: INIT adapter done
Jan 14 15:21:37 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 14 15:21:37 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 15:21:37 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 14 15:21:37 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 14 15:21:37 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 14 15:21:37 mds1 kernel: scsi1 : qla2xxx
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 14 15:21:37 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 14 15:21:37 mds1 kernel: scsi2 : qla2xxx
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 14 15:21:37 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 14 15:21:37 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 14 15:21:37 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 14 15:21:37 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 14 15:21:37 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 14 15:21:37 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 14 15:21:37 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 14 15:21:37 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 14 15:21:37 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2f8d, mb[3]=8100, cmd=6a ****.
Jan 14 15:21:37 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2fac, mb[3]=3100, cmd=6a ****.
Jan 14 15:21:37 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:21:37 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 14 15:21:37 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 14 15:21:37 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 14 15:21:37 mds1 kernel: scsi3 : ahci
Jan 14 15:21:37 mds1 kernel: scsi4 : ahci
Jan 14 15:21:37 mds1 kernel: scsi5 : ahci
Jan 14 15:21:37 mds1 kernel: scsi6 : ahci
Jan 14 15:21:37 mds1 kernel: scsi7 : ahci
Jan 14 15:21:37 mds1 kernel: scsi8 : ahci
Jan 14 15:21:37 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 14 15:21:37 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 14 15:21:37 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 14 15:21:37 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 14 15:21:37 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 14 15:21:37 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 14 15:21:37 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 14 15:21:37 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 14 15:21:37 mds1 kernel: ata1.00: configured for UDMA/100
Jan 14 15:21:37 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 14 15:21:37 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 14 15:21:37 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:21:37 mds1 kernel: sdb:
Jan 14 15:21:37 mds1 kernel: sdc:
Jan 14 15:21:37 mds1 kernel: sdd:
Jan 14 15:21:37 mds1 kernel: sdg: unknown partition table
Jan 14 15:21:37 mds1 kernel: sde: unknown partition table
Jan 14 15:21:37 mds1 kernel: sdf: unknown partition table
Jan 14 15:21:37 mds1 kernel: sda:
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: unknown partition table
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: unknown partition table
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: unknown partition table
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: sda1 sda2 sda3
Jan 14 15:21:37 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 14 15:21:37 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 14 15:21:37 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 14 15:21:37 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 14 15:21:37 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 14 15:21:37 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 14 15:21:37 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 14 15:21:37 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 14 15:21:37 mds1 kernel: SELinux:  Disabled at runtime.
Jan 14 15:21:37 mds1 kernel: type=1404 audit(1389730880.429:2): selinux=0 auid=4294967295 ses=4294967295
Jan 14 15:21:37 mds1 kernel: dracut: 
Jan 14 15:21:37 mds1 kernel: dracut: Switching root
Jan 14 15:21:37 mds1 kernel: udev: starting version 147
Jan 14 15:21:37 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 14 15:21:37 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 14 15:21:37 mds1 kernel: PTP clock support registered
Jan 14 15:21:37 mds1 kernel: dca service started, version 1.12.1
Jan 14 15:21:37 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 14 15:21:37 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:21:37 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 14 15:21:37 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 14 15:21:37 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 14 15:21:37 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:21:37 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:21:37 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 14 15:21:37 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 14 15:21:37 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 14 15:21:37 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 14 15:21:37 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 14 15:21:37 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 14 15:21:37 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:21:37 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 14 15:21:37 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 14 15:21:37 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 14 15:21:37 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:21:37 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:21:37 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 14 15:21:37 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:21:37 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 14 15:21:37 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 14 15:21:37 mds1 kernel: emc: device handler registered
Jan 14 15:21:37 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 15:21:37 mds1 kernel: device-mapper: table: 253:2: multipath: error getting device
Jan 14 15:21:37 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:21:37 mds1 kernel: device-mapper: table: 253:2: multipath: error getting device
Jan 14 15:21:37 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 15:21:37 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 14 15:21:37 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 14 15:21:37 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 14 15:21:37 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 14 15:21:37 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 14 15:21:37 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 14 15:21:37 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 14 15:21:37 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 14 15:21:37 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028787000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 14 15:21:37 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028787000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 14 15:21:37 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 14 15:21:37 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 14 15:21:37 mds1 kernel: NET: Registered protocol family 10
Jan 14 15:21:37 mds1 kernel: lo: Disabled Privacy Extensions
Jan 14 15:21:37 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 15:21:37 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 15:21:37 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 14 15:21:37 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 15:21:37 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:21:37 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 15:21:37 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:21:37 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 15:21:37 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 14 15:21:37 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 14 15:21:37 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 14 15:21:37 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 14 15:21:37 mds1 rpc.statd[3256]: Version 1.2.3 starting
Jan 14 15:21:37 mds1 sm-notify[3257]: Version 1.2.3 starting
Jan 14 15:21:37 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 14 15:21:37 mds1 kernel: RPC: Registered udp transport module.
Jan 14 15:21:37 mds1 kernel: RPC: Registered tcp transport module.
Jan 14 15:21:37 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 14 15:21:37 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 14 15:21:37 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 14 15:21:37 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 14 15:21:37 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 14 15:21:41 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 14 15:21:41 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 14 15:21:45 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 14 15:21:46 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 14 15:21:46 mds1 cib[3399]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:46 mds1 attrd[3402]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:46 mds1 stonith-ng[3400]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:46 mds1 pengine[3403]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:46 mds1 lrmd[3401]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:46 mds1 crmd[3404]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:21:47 mds1 logger[3413]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 14 15:21:48 mds1 kdump: kexec: loaded kdump kernel
Jan 14 15:21:48 mds1 kdump: started up
Jan 14 15:21:49 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 14 15:21:49 mds1 acpid: starting up
Jan 14 15:21:49 mds1 acpid: 1 rule loaded
Jan 14 15:21:49 mds1 acpid: waiting for events: event logging is off
Jan 14 15:21:49 mds1 acpid: client connected from 3617[68:68]
Jan 14 15:21:49 mds1 acpid: 1 client rule loaded
Jan 14 15:21:49 mds1 kernel: ipmi device interface
Jan 14 15:21:51 mds1 automount[3656]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 14 15:21:51 mds1 abrtd: Init complete, entering main loop
Jan 14 15:21:52 mds1 cimserver[3807]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 14 15:21:58 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 14 15:22:00 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 14 15:22:16 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 15:22:16 mds1 kernel: 
Jan 14 15:22:16 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 15:22:16 mds1 kernel: 
Jan 14 15:22:16 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 15:22:16 mds1 kernel: 
Jan 14 15:22:58 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 14 15:22:59 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 14 15:22:59 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:22:59 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:23:02 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 14 15:23:02 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:23:04 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389730979/real 0]  req@ffff880fe31e4c00 x1457238474489864/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389730984 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:23:04 mds1 kernel: LustreError: 4186:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 14 15:23:04 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 14 15:23:04 mds1 kernel: LustreError: 4403:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 15:23:04 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:23:04 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:23:04 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 15:23:04 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:23:04 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 15:23:04 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:23:04 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:23:04 mds1 kernel: LustreError: 4470:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 15:23:09 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389730984/real 0]  req@ffff880822e63000 x1457238474490260/t0(0) o8->emc-OST0003-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389730989 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:23:10 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389730985/real 0]  req@ffff880822efa800 x1457238474490268/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389730990 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:23:10 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Jan 14 15:23:30 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 15:23:30 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -19.
Jan 14 15:23:30 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 14 15:23:36 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389731010/real 0]  req@ffff880fcfc20400 x1457238474490800/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389731016 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:23:36 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 14 15:23:55 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 15:24:06 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389731035/real 0]  req@ffff880fcfbd4400 x1457238474491068/t0(0) o8->emc-OST0004-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731046 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:24:06 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 14 15:24:20 mds1 kernel: LustreError: 11-0: emc-OST0001-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -19.
Jan 14 15:24:20 mds1 kernel: LustreError: Skipped 23 previous similar messages
Jan 14 15:25:01 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389731085/real 0]  req@ffff880fcfc05800 x1457238474491480/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731101 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:25:01 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 14 15:25:10 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -19.
Jan 14 15:25:10 mds1 kernel: LustreError: Skipped 23 previous similar messages
Jan 14 15:25:21 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:25:21 mds1 kernel: LustreError: 5009:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0000: Aborting recovery
Jan 14 15:25:21 mds1 kernel: Lustre: 4524:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 14 15:25:21 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 1:26, of 5 clients 0 recovered and 0 were evicted.
Jan 14 15:25:21 mds1 kernel: Lustre: emc-MDT0001: Client emc-MDT0000-mdtlov_UUID (at 0@lo) reconnecting, waiting for 5 clients in recovery for 3:08
Jan 14 15:25:21 mds1 kernel: Lustre: emc-MDT0001: Client emc-MDT0000-mdtlov_UUID (at 0@lo) refused reconnection, still busy with 1 active RPCs
Jan 14 15:25:21 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -16.
Jan 14 15:25:21 mds1 kernel: LustreError: Skipped 23 previous similar messages
Jan 14 15:25:21 mds1 kernel: LustreError: 4473:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST000c-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 14 15:25:21 mds1 kernel: LustreError: 4499:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST000d-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 14 15:25:22 mds1 kernel: Lustre: server umount MGS complete
Jan 14 15:25:35 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation obd_ping failed with -107.
Jan 14 15:25:35 mds1 kernel: Lustre: emc-MDT0000-lwp-MDT0001: Connection to emc-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 15:25:35 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:25:45 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.46@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:25:45 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 14 15:25:46 mds1 kernel: Lustre: 3383:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731135/real 1389731135]  req@ffff880fe3283c00 x1457238474491848/t0(0) o400->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389731146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:25:46 mds1 kernel: Lustre: 3383:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 14 15:25:46 mds1 kernel: LustreError: 166-1: MGC192.168.1.40@o2ib: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Jan 14 15:26:03 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731035/real 1389731035]  req@ffff880fe5424000 x1457238474490968/t0(0) o400->emc-MDT0000-osp-MDT0001@0@lo:24/10 lens 224/224 e 4 to 1 dl 1389731163 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 14 15:26:03 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jan 14 15:26:03 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:26:03 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 14 15:26:11 mds1 kernel: LustreError: 11-0: emc-OST000a-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -19.
Jan 14 15:26:12 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:26:17 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:26:21 mds1 kernel: LustreError: 166-1: MGC192.168.1.40@o2ib: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Jan 14 15:26:21 mds1 kernel: Lustre: MGS: Client 80ed54d6-8974-f9df-f75e-763c5cc98230 (at 0@lo) reconnecting
Jan 14 15:26:21 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:26:21 mds1 kernel: LustreError: 5245:0:(genops.c:320:class_newdev()) Device emc-OST0006-osc-MDT0000 already exists at 41, won't add
Jan 14 15:26:21 mds1 kernel: LustreError: 5245:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST0006-osc-MDT0000 of type osp : -17
Jan 14 15:26:21 mds1 kernel: LustreError: 5245:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:26:21 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST0006-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 14 15:26:21 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:26:21 mds1 kernel: LustreError: 5132:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 14 15:26:21 mds1 kernel: LustreError: 5132:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:26:21 mds1 kernel: LustreError: 5132:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 14 15:26:21 mds1 kernel: LustreError: 5132:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 14 15:26:21 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:26:21 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 14 15:26:21 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 15:26:21 mds1 kernel: LustreError: 5132:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:26:27 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:26:27 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 14 15:26:46 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:26:52 mds1 kernel: Lustre: emc-MDT0001: Denying connection for new client f0262952-5ef1-b73f-7a21-9ab1ebd70a27 (at 192.168.1.55@o2ib), waiting for all 5 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:37
Jan 14 15:26:55 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:26:55 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:26:55 mds1 kernel: LustreError: 5519:0:(genops.c:320:class_newdev()) Device emc-OST0006-osc-MDT0000 already exists at 41, won't add
Jan 14 15:26:55 mds1 kernel: LustreError: 5519:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST0006-osc-MDT0000 of type osp : -17
Jan 14 15:26:55 mds1 kernel: LustreError: 5519:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:26:55 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST0006-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 14 15:26:55 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:26:55 mds1 kernel: LustreError: 5496:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 14 15:26:55 mds1 kernel: LustreError: 5496:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:26:55 mds1 kernel: LustreError: 5496:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 14 15:26:55 mds1 kernel: LustreError: 5496:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 14 15:26:55 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:26:56 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 14 15:26:56 mds1 kernel: LustreError: 5496:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:26:57 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731206/real 1389731206]  req@ffff880fd076f000 x1457238474492220/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389731217 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:26:57 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 14 15:27:15 mds1 kernel: INFO: task tgt_recov:4469 blocked for more than 120 seconds.
Jan 14 15:27:15 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 15:27:15 mds1 kernel: tgt_recov     D 0000000000000006     0  4469      2 0x00000080
Jan 14 15:27:15 mds1 kernel: ffff880fcfd85da0 0000000000000046 0000000000000000 ffff88085c4b6740
Jan 14 15:27:15 mds1 kernel: ffff880fcfd85d30 ffffffff81055f96 ffff880fcfd85d40 ffff881010e5b540
Jan 14 15:27:15 mds1 kernel: ffff880fcfd83ab8 ffff880fcfd85fd8 000000000000fb88 ffff880fcfd83ab8
Jan 14 15:27:15 mds1 kernel: Call Trace:
Jan 14 15:27:15 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 14 15:27:15 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:27:15 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 14 15:27:15 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 14 15:27:15 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 14 15:27:15 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 14 15:27:20 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:27:20 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 14 15:27:22 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:27:22 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:27:22 mds1 kernel: LustreError: 5776:0:(genops.c:320:class_newdev()) Device emc-OST0006-osc-MDT0000 already exists at 41, won't add
Jan 14 15:27:22 mds1 kernel: LustreError: 5776:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST0006-osc-MDT0000 of type osp : -17
Jan 14 15:27:22 mds1 kernel: LustreError: 5776:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:27:22 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST0006-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 14 15:27:22 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:27:22 mds1 kernel: LustreError: 5752:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 14 15:27:22 mds1 kernel: LustreError: 5752:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:27:22 mds1 kernel: LustreError: 5752:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 14 15:27:22 mds1 kernel: LustreError: 5752:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 14 15:27:22 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:27:22 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 14 15:27:22 mds1 kernel: LustreError: 5752:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:27:27 mds1 kernel: Lustre: emc-OST000f-osc-MDT0001: Connection to emc-OST000f (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 15:27:43 mds1 cibadmin[5824]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 15:27:43 mds1 cibadmin[5825]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 15:28:08 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731272/real 1389731272]  req@ffff880fcfd9c000 x1457238474492644/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389731288 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:28:08 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
Jan 14 15:28:10 mds1 cibadmin[5856]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 15:28:10 mds1 cibadmin[5857]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 15:28:11 mds1 cibadmin[5864]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 15:28:11 mds1 cibadmin[5865]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 15:28:17 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:28:17 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 14 15:28:30 mds1 kernel: Lustre: emc-MDT0001: recovery is timed out, evict stale exports
Jan 14 15:28:30 mds1 kernel: Lustre: emc-MDT0001: disconnecting 4 stale clients
Jan 14 15:28:30 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 5:00, of 5 clients 1 recovered and 4 were evicted.
Jan 14 15:28:56 mds1 cibadmin[5944]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 15:28:56 mds1 cibadmin[5945]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 15:29:22 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:29:22 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:29:22 mds1 kernel: LustreError: 6044:0:(genops.c:320:class_newdev()) Device emc-OST0006-osc-MDT0000 already exists at 41, won't add
Jan 14 15:29:22 mds1 kernel: LustreError: 6044:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST0006-osc-MDT0000 of type osp : -17
Jan 14 15:29:22 mds1 kernel: LustreError: 6044:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:29:22 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST0006-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 14 15:29:22 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:29:22 mds1 kernel: LustreError: 6021:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 14 15:29:22 mds1 kernel: LustreError: 6021:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:29:22 mds1 kernel: LustreError: 6021:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 14 15:29:22 mds1 kernel: LustreError: 6021:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 14 15:29:22 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:29:23 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 14 15:29:23 mds1 kernel: LustreError: 6021:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:29:36 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.42@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:29:36 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 14 15:29:49 mds1 cibadmin[6076]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 15:29:49 mds1 cibadmin[6077]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 15:30:04 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:30:04 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:30:04 mds1 kernel: LustreError: 6116:0:(genops.c:320:class_newdev()) Device emc-OST000b-osc-MDT0000 already exists at 45, won't add
Jan 14 15:30:04 mds1 kernel: LustreError: 6116:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST000b-osc-MDT0000 of type osp : -17
Jan 14 15:30:04 mds1 kernel: LustreError: 6116:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:30:04 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST000b-osc-MDT0000  1:osp  2:emc-MDT0000-mdtlov_UUID  
Jan 14 15:30:04 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:30:04 mds1 kernel: LustreError: 6092:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0000: -17
Jan 14 15:30:04 mds1 kernel: LustreError: 6092:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:30:04 mds1 kernel: LustreError: 6092:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0000: Can't end config log emc-client.
Jan 14 15:30:04 mds1 kernel: LustreError: 6092:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0000: failed to disconnect lwp. (rc=-2)
Jan 14 15:30:04 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 15:30:05 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 14 15:30:05 mds1 kernel: LustreError: 6092:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:30:40 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731404/real 1389731404]  req@ffff880feeb13800 x1457238474493292/t0(0) o8->emc-OST0003-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731440 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:30:40 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 88 previous similar messages
Jan 14 15:31:44 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:31:44 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 14 15:32:22 mds1 init: tty (/dev/tty1) main process (3832) killed by TERM signal
Jan 14 15:32:22 mds1 init: tty (/dev/tty2) main process (3834) killed by TERM signal
Jan 14 15:32:22 mds1 init: tty (/dev/tty3) main process (3836) killed by TERM signal
Jan 14 15:32:22 mds1 init: tty (/dev/tty4) main process (3838) killed by TERM signal
Jan 14 15:32:22 mds1 init: tty (/dev/tty5) main process (3840) killed by TERM signal
Jan 14 15:32:22 mds1 init: tty (/dev/tty6) main process (3842) killed by TERM signal
Jan 14 15:32:22 mds1 abrtd: Got signal 15, exiting
Jan 14 15:32:26 mds1 acpid: exiting
Jan 14 15:32:26 mds1 multipathd: mdt1: event checker exit
Jan 14 15:32:26 mds1 multipathd: dm-1: remove map (uevent)
Jan 14 15:32:26 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 14 15:32:26 mds1 kernel: sd 1:0:0:1: emc: Detached
Jan 14 15:32:26 mds1 kernel: sd 2:0:0:1: emc: Detached
Jan 14 15:32:26 mds1 multipathd: dm-1: remove map (uevent)
Jan 14 15:32:26 mds1 multipathd: dm-1: devmap not registered, can't remove
Jan 14 15:33:21 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 14 15:33:22 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 14 15:34:53 mds1 init: Disconnected from system bus
Jan 14 15:34:53 mds1 multipathd: mgt: stop event checker thread (140257910048512)
Jan 14 15:34:53 mds1 multipathd: mdt2: stop event checker thread (140257899271936)
Jan 14 15:34:53 mds1 multipathd: --------shut down-------
Jan 14 15:34:53 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 14 15:34:53 mds1 auditd[3136]: The audit daemon is exiting.
Jan 14 15:34:53 mds1 kernel: type=1305 audit(1389731693.647:60): audit_pid=0 old=3136 auid=4294967295 ses=4294967295 res=1
Jan 14 15:34:53 mds1 kernel: type=1305 audit(1389731693.747:61): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Jan 14 15:34:53 mds1 kernel: Kernel logging (proc) stopped.
Jan 14 15:34:53 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3161" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 14 15:37:40 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 14 15:37:40 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3097" x-info="http://www.rsyslog.com"] start
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys cpuset
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys cpu
Jan 14 15:37:40 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 14 15:37:40 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 15:37:40 mds1 kernel: KERNEL supported cpus:
Jan 14 15:37:40 mds1 kernel:  Intel GenuineIntel
Jan 14 15:37:40 mds1 kernel:  AMD AuthenticAMD
Jan 14 15:37:40 mds1 kernel:  Centaur CentaurHauls
Jan 14 15:37:40 mds1 kernel: BIOS-provided physical RAM map:
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 14 15:37:40 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 14 15:37:40 mds1 kernel: DMI 2.6 present.
Jan 14 15:37:40 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 14 15:37:40 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 14 15:37:40 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 14 15:37:40 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 14 15:37:40 mds1 kernel: Using GB pages for direct mapping
Jan 14 15:37:40 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 14 15:37:40 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 14 15:37:40 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 14 15:37:40 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 14 15:37:40 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 14 15:37:40 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 14 15:37:40 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 15:37:40 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:37:40 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:37:40 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:37:40 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 15:37:40 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 14 15:37:40 mds1 kernel: Setting APIC routing to flat.
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 14 15:37:40 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 14 15:37:40 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 14 15:37:40 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 14 15:37:40 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 14 15:37:40 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 14 15:37:40 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 14 15:37:40 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 14 15:37:40 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 14 15:37:40 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 14 15:37:40 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 14 15:37:40 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 14 15:37:40 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 14 15:37:40 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 14 15:37:40 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 14 15:37:40 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 14 15:37:40 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 14 15:37:40 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 14 15:37:40 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 14 15:37:40 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 14 15:37:40 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 14 15:37:40 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 14 15:37:40 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 14 15:37:40 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 14 15:37:40 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 14 15:37:40 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 14 15:37:40 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 14 15:37:40 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 14 15:37:40 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 14 15:37:40 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 14 15:37:40 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 14 15:37:40 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 14 15:37:40 mds1 kernel: Zone PFN ranges:
Jan 14 15:37:40 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 14 15:37:40 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 14 15:37:40 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 14 15:37:40 mds1 kernel: Movable zone start PFN for each node
Jan 14 15:37:40 mds1 kernel: early_node_map[5] active PFN ranges
Jan 14 15:37:40 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 14 15:37:40 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 14 15:37:40 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 14 15:37:40 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 14 15:37:40 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 14 15:37:40 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 14 15:37:40 mds1 kernel: Setting APIC routing to flat.
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 14 15:37:40 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 14 15:37:40 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 14 15:37:40 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 14 15:37:40 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 14 15:37:40 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 14 15:37:40 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 14 15:37:40 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 14 15:37:40 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 14 15:37:40 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 14 15:37:40 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 14 15:37:40 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 14 15:37:40 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 14 15:37:40 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 14 15:37:40 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 14 15:37:40 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 14 15:37:40 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 14 15:37:40 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 14 15:37:40 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 14 15:37:40 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 14 15:37:40 mds1 kernel: Policy zone: Normal
Jan 14 15:37:40 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 15:37:40 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 14 15:37:40 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 14 15:37:40 mds1 kernel: Checking aperture...
Jan 14 15:37:40 mds1 kernel: No AGP bridge found
Jan 14 15:37:40 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 14 15:37:40 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 14 15:37:40 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 14 15:37:40 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 14 15:37:40 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 14 15:37:40 mds1 kernel: Hierarchical RCU implementation.
Jan 14 15:37:40 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 14 15:37:40 mds1 kernel: Extended CMOS year: 2000
Jan 14 15:37:40 mds1 kernel: Console: colour VGA+ 80x25
Jan 14 15:37:40 mds1 kernel: console [tty0] enabled
Jan 14 15:37:40 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 14 15:37:40 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 14 15:37:40 mds1 kernel: Fast TSC calibration using PIT
Jan 14 15:37:40 mds1 kernel: Detected 1995.258 MHz processor.
Jan 14 15:37:40 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.51 BogoMIPS (lpj=1995258)
Jan 14 15:37:40 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 14 15:37:40 mds1 kernel: Security Framework initialized
Jan 14 15:37:40 mds1 kernel: SELinux:  Initializing.
Jan 14 15:37:40 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 14 15:37:40 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 14 15:37:40 mds1 kernel: Mount-cache hash table entries: 256
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys ns
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys memory
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys devices
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys freezer
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys net_cls
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys blkio
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys perf_event
Jan 14 15:37:40 mds1 kernel: Initializing cgroup subsys net_prio
Jan 14 15:37:40 mds1 kernel: CPU: Physical Processor ID: 0
Jan 14 15:37:40 mds1 kernel: CPU: Processor Core ID: 0
Jan 14 15:37:40 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 14 15:37:40 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 14 15:37:40 mds1 kernel: using mwait in idle threads.
Jan 14 15:37:40 mds1 kernel: ACPI: Core revision 20090903
Jan 14 15:37:40 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 14 15:37:40 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 14 15:37:40 mds1 kernel: dmar: Host address width 46
Jan 14 15:37:40 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 14 15:37:40 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 15:37:40 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 14 15:37:40 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 15:37:40 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 14 15:37:40 mds1 kernel: dmar: ATSR flags: 0x0
Jan 14 15:37:40 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 14 15:37:40 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 14 15:37:40 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 14 15:37:40 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 14 15:37:40 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 14 15:37:40 mds1 kernel: Enabling x2apic
Jan 14 15:37:40 mds1 kernel: Enabled x2apic
Jan 14 15:37:40 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 14 15:37:40 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 14 15:37:40 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 14 15:37:40 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 14 15:37:40 mds1 kernel: PEBS disabled due to CPU errata.
Jan 14 15:37:40 mds1 kernel: ... version:                3
Jan 14 15:37:40 mds1 kernel: ... bit width:              48
Jan 14 15:37:40 mds1 kernel: ... generic registers:      8
Jan 14 15:37:40 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 14 15:37:40 mds1 kernel: ... max period:             000000007fffffff
Jan 14 15:37:40 mds1 kernel: ... fixed-purpose events:   3
Jan 14 15:37:40 mds1 kernel: ... event mask:             00000007000000ff
Jan 14 15:37:40 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 14 15:37:40 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 14 15:37:40 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 14 15:37:40 mds1 kernel: Brought up 12 CPUs
Jan 14 15:37:40 mds1 kernel: Total of 12 processors activated (47882.16 BogoMIPS).
Jan 14 15:37:40 mds1 kernel: devtmpfs: initialized
Jan 14 15:37:40 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 14 15:37:40 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 14 15:37:40 mds1 kernel: regulator: core version 0.5
Jan 14 15:37:40 mds1 kernel: NET: Registered protocol family 16
Jan 14 15:37:40 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 14 15:37:40 mds1 kernel: ACPI: bus type pci registered
Jan 14 15:37:40 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 14 15:37:40 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 14 15:37:40 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 14 15:37:40 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 14 15:37:40 mds1 kernel: bio: create slab <bio-0> at 0
Jan 14 15:37:40 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 14 15:37:40 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 14 15:37:40 mds1 kernel: ACPI: Interpreter enabled
Jan 14 15:37:40 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 14 15:37:40 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 14 15:37:40 mds1 kernel: ACPI: No dock devices found.
Jan 14 15:37:40 mds1 kernel: HEST: Table parsing has been initialized.
Jan 14 15:37:40 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 14 15:37:40 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 14 15:37:40 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 14 15:37:40 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 14 15:37:40 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:37:40 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 14 15:37:40 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 14 15:37:40 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 14 15:37:40 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 14 15:37:40 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 14 15:37:40 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:37:40 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 15:37:40 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 14 15:37:40 mds1 kernel: vgaarb: loaded
Jan 14 15:37:40 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 14 15:37:40 mds1 kernel: SCSI subsystem initialized
Jan 14 15:37:40 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 14 15:37:40 mds1 kernel: usbcore: registered new interface driver hub
Jan 14 15:37:40 mds1 kernel: usbcore: registered new device driver usb
Jan 14 15:37:40 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 14 15:37:40 mds1 kernel: NetLabel: Initializing
Jan 14 15:37:40 mds1 kernel: NetLabel:  domain hash size = 128
Jan 14 15:37:40 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 14 15:37:40 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 14 15:37:40 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 14 15:37:40 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 14 15:37:40 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 14 15:37:40 mds1 kernel: Switching to clocksource hpet
Jan 14 15:37:40 mds1 kernel: pnp: PnP ACPI init
Jan 14 15:37:40 mds1 kernel: ACPI: bus type pnp registered
Jan 14 15:37:40 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 14 15:37:40 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 14 15:37:40 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 14 15:37:40 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:37:40 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:37:40 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 15:37:40 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:37:40 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 14 15:37:40 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:37:40 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:37:40 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 15:37:40 mds1 kernel: NET: Registered protocol family 2
Jan 14 15:37:40 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 14 15:37:40 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 14 15:37:40 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 14 15:37:40 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 14 15:37:40 mds1 kernel: TCP reno registered
Jan 14 15:37:40 mds1 kernel: NET: Registered protocol family 1
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 15:37:40 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 14 15:37:40 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 14 15:37:40 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 14 15:37:40 mds1 kernel: dmar: Unsupported device scope
Jan 14 15:37:40 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 14 15:37:40 mds1 kernel: type=2000 audit(1389731839.054:1): initialized
Jan 14 15:37:40 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 14 15:37:40 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 14 15:37:40 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 14 15:37:40 mds1 kernel: msgmni has been set to 32768
Jan 14 15:37:40 mds1 kernel: alg: No test for stdrng (krng)
Jan 14 15:37:40 mds1 kernel: ksign: Installing public key data
Jan 14 15:37:40 mds1 kernel: Loading keyring
Jan 14 15:37:40 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 14 15:37:40 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 14 15:37:40 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 14 15:37:40 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 14 15:37:40 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 14 15:37:40 mds1 kernel: io scheduler noop registered
Jan 14 15:37:40 mds1 kernel: io scheduler anticipatory registered
Jan 14 15:37:40 mds1 kernel: io scheduler deadline registered (default)
Jan 14 15:37:40 mds1 kernel: io scheduler cfq registered
Jan 14 15:37:40 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 14 15:37:40 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 14 15:37:40 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 14 15:37:40 mds1 kernel: acpiphp: Slot [2] registered
Jan 14 15:37:40 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 14 15:37:40 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 14 15:37:40 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 14 15:37:40 mds1 kernel: ipmi message handler version 39.2
Jan 14 15:37:40 mds1 kernel: IPMI System Interface driver.
Jan 14 15:37:40 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 14 15:37:40 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 14 15:37:40 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 14 15:37:40 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 14 15:37:40 mds1 kernel: IPMI kcs interface initialized
Jan 14 15:37:40 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 14 15:37:40 mds1 kernel: ACPI: Power Button [PWRF]
Jan 14 15:37:40 mds1 kernel: ERST: Could not register with persistent store
Jan 14 15:37:40 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 14 15:37:40 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 14 15:37:40 mds1 kernel: Non-volatile memory driver v1.3
Jan 14 15:37:40 mds1 kernel: Linux agpgart interface v0.103
Jan 14 15:37:40 mds1 kernel: crash memory driver: version 1.1
Jan 14 15:37:40 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 14 15:37:40 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 15:37:40 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 15:37:40 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 15:37:40 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 15:37:40 mds1 kernel: brd: module loaded
Jan 14 15:37:40 mds1 kernel: loop: module loaded
Jan 14 15:37:40 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 14 15:37:40 mds1 kernel: Fixed MDIO Bus: probed
Jan 14 15:37:40 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 14 15:37:40 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 15:37:40 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 15:37:40 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 14 15:37:40 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 15:37:40 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 14 15:37:40 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 14 15:37:40 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 14 15:37:40 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 14 15:37:40 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 15:37:40 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 15:37:40 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 14 15:37:40 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 15:37:40 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 14 15:37:40 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 14 15:37:40 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 14 15:37:40 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 14 15:37:40 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 14 15:37:40 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 14 15:37:40 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 14 15:37:40 mds1 kernel: Switching to clocksource tsc
Jan 14 15:37:40 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 14 15:37:40 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 14 15:37:40 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 14 15:37:40 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 14 15:37:40 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 14 15:37:40 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 14 15:37:40 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 14 15:37:40 mds1 kernel: cpuidle: using governor ladder
Jan 14 15:37:40 mds1 kernel: cpuidle: using governor menu
Jan 14 15:37:40 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 14 15:37:40 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 14 15:37:40 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 14 15:37:40 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 14 15:37:40 mds1 kernel: TCP cubic registered
Jan 14 15:37:40 mds1 kernel: Initializing XFRM netlink socket
Jan 14 15:37:40 mds1 kernel: NET: Registered protocol family 17
Jan 14 15:37:40 mds1 kernel: registered taskstats version 1
Jan 14 15:37:40 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-14 20:37:21 UTC (1389731841)
Jan 14 15:37:40 mds1 kernel: Initalizing network drop monitor service
Jan 14 15:37:40 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 14 15:37:40 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 14 15:37:40 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 14 15:37:40 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 14 15:37:40 mds1 kernel: dracut: dracut-004-303.el6
Jan 14 15:37:40 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 14 15:37:40 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 14 15:37:40 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 14 15:37:40 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 14 15:37:40 mds1 kernel: udev: starting version 147
Jan 14 15:37:40 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 15:37:40 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 15:37:40 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 14 15:37:40 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 14 15:37:40 mds1 kernel: dracut: Starting plymouth daemon
Jan 14 15:37:40 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 14 15:37:40 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 14 15:37:40 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 14 15:37:40 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 14 15:37:40 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 14 15:37:40 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 14 15:37:40 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 14 15:37:40 mds1 kernel: megasas: FW now in Ready state
Jan 14 15:37:40 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 15:37:40 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 15:37:40 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 14 15:37:40 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 14 15:37:40 mds1 kernel: megasas:IOC Init cmd success
Jan 14 15:37:40 mds1 kernel: megasas: INIT adapter done
Jan 14 15:37:40 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 14 15:37:40 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 15:37:40 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 14 15:37:40 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 14 15:37:40 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 14 15:37:40 mds1 kernel: scsi1 : qla2xxx
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 14 15:37:40 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 14 15:37:40 mds1 kernel: scsi2 : qla2xxx
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 14 15:37:40 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 14 15:37:40 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 14 15:37:40 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 14 15:37:40 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 14 15:37:40 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 14 15:37:40 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 14 15:37:40 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 14 15:37:40 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 14 15:37:40 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2de8, mb[3]=8100, cmd=6a ****.
Jan 14 15:37:40 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dc9, mb[3]=4100, cmd=6a ****.
Jan 14 15:37:40 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 15:37:40 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 14 15:37:40 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 14 15:37:40 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 14 15:37:40 mds1 kernel: scsi3 : ahci
Jan 14 15:37:40 mds1 kernel: scsi4 : ahci
Jan 14 15:37:40 mds1 kernel: scsi5 : ahci
Jan 14 15:37:40 mds1 kernel: scsi6 : ahci
Jan 14 15:37:40 mds1 kernel: scsi7 : ahci
Jan 14 15:37:40 mds1 kernel: scsi8 : ahci
Jan 14 15:37:40 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 14 15:37:40 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 14 15:37:40 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 14 15:37:40 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 14 15:37:40 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 14 15:37:40 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 14 15:37:40 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 14 15:37:40 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 14 15:37:40 mds1 kernel: ata1.00: configured for UDMA/100
Jan 14 15:37:40 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 14 15:37:40 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 14 15:37:40 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sda:
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 14 15:37:40 mds1 kernel: sdb:
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 15:37:40 mds1 kernel: sdc: unknown partition table
Jan 14 15:37:40 mds1 kernel: unknown partition table
Jan 14 15:37:40 mds1 kernel: sdd:
Jan 14 15:37:40 mds1 kernel: sde:
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sdg:
Jan 14 15:37:40 mds1 kernel: sdf:
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: unknown partition table
Jan 14 15:37:40 mds1 kernel: unknown partition table
Jan 14 15:37:40 mds1 kernel: unknown partition table
Jan 14 15:37:40 mds1 kernel: unknown partition table
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sda1 sda2 sda3
Jan 14 15:37:40 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 14 15:37:40 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 14 15:37:40 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 14 15:37:40 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 14 15:37:40 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 14 15:37:40 mds1 kernel: SELinux:  Disabled at runtime.
Jan 14 15:37:40 mds1 kernel: type=1404 audit(1389731844.199:2): selinux=0 auid=4294967295 ses=4294967295
Jan 14 15:37:40 mds1 kernel: dracut: 
Jan 14 15:37:40 mds1 kernel: dracut: Switching root
Jan 14 15:37:40 mds1 kernel: readahead: starting
Jan 14 15:37:40 mds1 kernel: udev: starting version 147
Jan 14 15:37:40 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 14 15:37:40 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 14 15:37:40 mds1 kernel: PTP clock support registered
Jan 14 15:37:40 mds1 kernel: dca service started, version 1.12.1
Jan 14 15:37:40 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 14 15:37:40 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 15:37:40 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 14 15:37:40 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 14 15:37:40 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 14 15:37:40 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 15:37:40 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 15:37:40 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 14 15:37:40 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 14 15:37:40 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 14 15:37:40 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 14 15:37:40 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 14 15:37:40 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 14 15:37:40 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 15:37:40 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 14 15:37:40 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 14 15:37:40 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 14 15:37:40 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 15:37:40 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 15:37:40 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 14 15:37:40 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 14 15:37:40 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 14 15:37:40 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 14 15:37:40 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 14 15:37:40 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 14 15:37:40 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:37:40 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 14 15:37:40 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:37:40 mds1 kernel: emc: device handler registered
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 14 15:37:40 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 14 15:37:40 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 14 15:37:40 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 14 15:37:40 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 14 15:37:40 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 14 15:37:40 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 14 15:37:40 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 14 15:37:40 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028c9c000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 14 15:37:40 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028c9c000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 14 15:37:40 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 14 15:37:40 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 14 15:37:40 mds1 kernel: NET: Registered protocol family 10
Jan 14 15:37:40 mds1 kernel: lo: Disabled Privacy Extensions
Jan 14 15:37:40 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 15:37:40 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 15:37:40 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 14 15:37:40 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 15:37:40 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:37:40 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 15:37:40 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 15:37:40 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 15:37:40 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 14 15:37:40 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 14 15:37:40 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 14 15:37:40 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 14 15:37:41 mds1 rpc.statd[3192]: Version 1.2.3 starting
Jan 14 15:37:41 mds1 sm-notify[3193]: Version 1.2.3 starting
Jan 14 15:37:41 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 14 15:37:41 mds1 kernel: RPC: Registered udp transport module.
Jan 14 15:37:41 mds1 kernel: RPC: Registered tcp transport module.
Jan 14 15:37:41 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 14 15:37:41 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 14 15:37:41 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 14 15:37:41 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 14 15:37:41 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 14 15:37:45 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 14 15:37:45 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 14 15:37:49 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 14 15:37:49 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 14 15:37:50 mds1 cib[3335]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 attrd[3338]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 pengine[3339]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 crmd[3340]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 stonith-ng[3336]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 lrmd[3337]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 15:37:50 mds1 logger[3350]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 14 15:37:52 mds1 kdump: kexec: loaded kdump kernel
Jan 14 15:37:52 mds1 kdump: started up
Jan 14 15:37:54 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 14 15:37:54 mds1 acpid: starting up
Jan 14 15:37:54 mds1 acpid: 1 rule loaded
Jan 14 15:37:54 mds1 acpid: waiting for events: event logging is off
Jan 14 15:37:55 mds1 acpid: client connected from 3558[68:68]
Jan 14 15:37:55 mds1 acpid: 1 client rule loaded
Jan 14 15:37:55 mds1 kernel: ipmi device interface
Jan 14 15:37:56 mds1 automount[3602]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 14 15:37:56 mds1 abrtd: Init complete, entering main loop
Jan 14 15:37:57 mds1 cimserver[3874]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 14 15:38:01 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 14 15:38:03 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 14 15:38:10 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:38:15 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731890/real 1389731890]  req@ffff880ffd5dbc00 x1457239484268548/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389731895 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:38:21 mds1 kernel: LustreError: 4035:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ffd5db800 x1457239484268552/t0(0) o253->MGC192.168.1.40@o2ib@0@lo:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 14 15:38:21 mds1 kernel: LustreError: 4035:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 14 15:38:25 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:38:25 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 14 15:38:25 mds1 kernel: LustreError: 4214:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 15:38:25 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:38:25 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:38:26 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:38:30 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731905/real 1389731905]  req@ffff88082290c000 x1457239484268664/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731910 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:38:30 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 15:38:30 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:38:30 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:38:30 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 15:38:30 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 15:38:30 mds1 kernel: LustreError: 4369:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 15:38:34 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 14 15:38:34 mds1 kernel: LustreError: 4595:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 14 15:38:34 mds1 kernel: Lustre: 4366:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 14 15:38:34 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 14 15:38:35 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731910/real 1389731910]  req@ffff880fe8bf3c00 x1457239484268868/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731915 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:38:35 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 14 15:38:38 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:38:38 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 14 15:38:38 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 14 15:38:38 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 14 15:38:38 mds1 kernel: LustreError: 4710:0:(genops.c:320:class_newdev()) Device emc-OST0006-osc-MDT0001 already exists at 9, won't add
Jan 14 15:38:38 mds1 kernel: LustreError: 4710:0:(obd_config.c:374:class_attach()) Cannot create device emc-OST0006-osc-MDT0001 of type osp : -17
Jan 14 15:38:38 mds1 kernel: LustreError: 4710:0:(obd_config.c:1591:class_config_llog_handler()) MGC192.168.1.40@o2ib: cfg command failed: rc = -17
Jan 14 15:38:38 mds1 kernel: Lustre:    cmd=cf001 0:emc-OST0006-osc-MDT0001  1:osp  2:emc-MDT0001-mdtlov_UUID  
Jan 14 15:38:38 mds1 kernel: LustreError: 15c-8: MGC192.168.1.40@o2ib: The configuration from log 'emc-MDT0001' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Jan 14 15:38:38 mds1 kernel: LustreError: 4668:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server emc-MDT0001: -17
Jan 14 15:38:38 mds1 kernel: LustreError: 4668:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -17
Jan 14 15:38:38 mds1 kernel: LustreError: 4668:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) emc-MDT0000-lwp-MDT0001: Can't end config log emc-client.
Jan 14 15:38:38 mds1 kernel: LustreError: 4668:0:(obd_mount_server.c:1426:server_put_super()) emc-MDT0001: failed to disconnect lwp. (rc=-2)
Jan 14 15:38:38 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 14 15:38:39 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 14 15:38:39 mds1 kernel: LustreError: 4668:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount  (-17)
Jan 14 15:39:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:39:03 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:39:08 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389731943/real 0]  req@ffff8810115ae800 x1457239484269216/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731948 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:39:08 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 14 15:39:38 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389731968/real 1389731968]  req@ffff881032dc2800 x1457239484269324/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389731978 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:39:38 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 14 15:39:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:39:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:40:03 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389731993/real 0]  req@ffff880823376400 x1457239484269432/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389732003 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:40:03 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 14 15:40:33 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389732018/real 1389732018]  req@ffff88082e72dc00 x1457239484269540/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389732033 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:40:33 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 14 15:40:43 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:40:43 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 14 15:40:58 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389732043/real 0]  req@ffff88083178cc00 x1457239484269648/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389732058 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:40:58 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 14 15:41:20 mds1 kernel: INFO: task tgt_recov:4422 blocked for more than 120 seconds.
Jan 14 15:41:20 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 15:41:20 mds1 kernel: tgt_recov     D 0000000000000001     0  4422      2 0x00000080
Jan 14 15:41:20 mds1 kernel: ffff880fe830fda0 0000000000000046 0000000000000000 ffff88085c496740
Jan 14 15:41:20 mds1 kernel: ffff880fe830fd30 ffffffff81055f96 ffff880fe830fd40 ffff881030464aa0
Jan 14 15:41:20 mds1 kernel: ffff880fe830dab8 ffff880fe830ffd8 000000000000fb88 ffff880fe830dab8
Jan 14 15:41:20 mds1 kernel: Call Trace:
Jan 14 15:41:20 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 14 15:41:20 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:41:20 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 14 15:41:20 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 14 15:41:20 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 14 15:41:20 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 14 15:41:27 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 15:41:27 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 14 15:41:27 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 14 15:41:27 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 15:41:27 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 14 15:41:32 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389732087/real 1389732087]  req@ffff880fe5edf000 x1457239484269932/t0(0) o8->emc-OST0017-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389732092 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:41:32 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 48 previous similar messages
Jan 14 15:41:52 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 14 15:41:52 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 14 15:41:52 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 14 15:42:17 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 15:42:23 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 14 15:42:23 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID (at 0@lo) reconnecting, waiting for 5 clients in recovery for 4:53
Jan 14 15:42:23 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID (at 0@lo) refused reconnection, still busy with 1 active RPCs
Jan 14 15:42:23 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -16.
Jan 14 15:42:24 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 14 15:42:25 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:42:42 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389732137/real 1389732162]  req@ffff880822ddf400 x1457239484270352/t0(0) o8->emc-OST000e-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389732162 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 15:42:42 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
Jan 14 15:42:42 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 15:42:42 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 15:43:07 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 14 15:43:15 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:44:05 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:44:55 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:45:20 mds1 kernel: INFO: task tgt_recov:4422 blocked for more than 120 seconds.
Jan 14 15:45:20 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 15:45:20 mds1 kernel: tgt_recov     D 0000000000000003     0  4422      2 0x00000080
Jan 14 15:45:20 mds1 kernel: ffff880fe830fda0 0000000000000046 0000000000000000 ffff88085c496740
Jan 14 15:45:20 mds1 kernel: ffff880fe830fd30 ffffffff81055f96 ffff880fe830fd40 ffff881030464aa0
Jan 14 15:45:20 mds1 kernel: ffff880fe830dab8 ffff880fe830ffd8 000000000000fb88 ffff880fe830dab8
Jan 14 15:45:20 mds1 kernel: Call Trace:
Jan 14 15:45:20 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 14 15:45:20 mds1 kernel: [<ffffffff8109708e>] ? prepare_to_wait+0x4e/0x80
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 14 15:45:20 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 15:45:20 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 14 15:45:20 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 14 15:45:20 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 14 15:45:20 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 14 15:45:45 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:46:35 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:47:17 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 14 15:47:17 mds1 kernel: Lustre: emc-MDT0000: disconnecting 4 stale clients
Jan 14 15:47:17 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 5:00, of 5 clients 1 recovered and 4 were evicted.
Jan 14 15:47:25 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:48:15 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 14 15:48:29 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection restored to emc-OST0006 (at 192.168.1.44@o2ib)
Jan 14 15:48:49 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0001-mdtlov_UUID (at 0@lo) in 386 seconds. I think it's dead, and I am evicting it. exp ffff880fe8c03000, cur 1389732529 expire 1389732379 last 1389732143
Jan 14 15:50:31 mds1 kernel: Lustre: 3317:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389732607/real 1389732631]  req@ffff8808233afc00 x1457239484274432/t0(0) o13->emc-OST0011-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389732633 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 14 15:50:31 mds1 kernel: Lustre: 3317:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
Jan 14 15:50:31 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 15:51:21 mds1 kernel: Lustre: MGS: Client 9fea9d87-633b-5f23-4ad9-3b9b6268a34e (at 192.168.1.46@o2ib) reconnecting
Jan 14 15:51:21 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0017_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 14 15:51:28 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 14 15:51:28 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 15:53:03 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0017_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 14 15:53:03 mds1 kernel: Lustre: Skipped 47 previous similar messages
Jan 14 15:53:53 mds1 kernel: Lustre: MGS: haven't heard from client c51e6be2-2705-141c-6a25-d0a887016657 (at 192.168.1.44@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8810115a8000, cur 1389732833 expire 1389732683 last 1389732604
Jan 14 15:54:06 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0011_UUID (at 192.168.1.44@o2ib) in 242 seconds. I think it's dead, and I am evicting it. exp ffff88083109d400, cur 1389732846 expire 1389732696 last 1389732604
Jan 14 16:37:58 mds1 kernel: LustreError: 11-0: emc-OST0005-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 14 16:37:58 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 16:37:58 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 14 16:37:58 mds1 kernel: LustreError: Skipped 14 previous similar messages
Jan 14 16:37:58 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection to emc-OST000a (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 16:37:58 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 14 16:38:04 mds1 kernel: Lustre: MGS: Client 55886b7d-322d-8035-4cf3-5097055f187f (at 192.168.1.44@o2ib) reconnecting
Jan 14 16:38:12 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.44@o2ib)
Jan 14 16:38:12 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 14 16:46:39 mds1 kernel: Lustre: 3314:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389735992/real 0]  req@ffff881019538c00 x1457239484341136/t0(0) o13->emc-OST000a-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389735999 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 14 16:46:39 mds1 kernel: Lustre: 3314:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 71 previous similar messages
Jan 14 16:46:39 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection to emc-OST000a (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 16:46:39 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 14 16:47:37 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 14 16:47:37 mds1 kernel: Lustre: Skipped 35 previous similar messages
Jan 14 16:47:40 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389736049/real 0]  req@ffff880fe551e400 x1457239484341904/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389736060 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 16:47:40 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 14 16:47:46 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection restored to emc-OST0003 (at 192.168.1.46@o2ib)
Jan 14 16:47:46 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 16:48:27 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 14 16:48:27 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 16:49:17 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0006_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 14 16:49:17 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 16:50:32 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0007_UUID (at 192.168.1.44@o2ib) in 247 seconds. I think it's dead, and I am evicting it. exp ffff880826e7e400, cur 1389736232 expire 1389736082 last 1389735985
Jan 14 16:50:32 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 14 16:58:06 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 14 16:58:06 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection to emc-OST000a (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 16:58:06 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 16:58:06 mds1 kernel: LustreError: Skipped 17 previous similar messages
Jan 14 16:58:08 mds1 kernel: LustreError: 11-0: emc-OST000c-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 14 16:58:08 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 16:58:08 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 14 16:58:08 mds1 kernel: LustreError: Skipped 6 previous similar messages
Jan 14 16:58:19 mds1 kernel: Lustre: MGS: Client 877567e0-15d6-41b3-9882-6d87f873074f (at 192.168.1.44@o2ib) reconnecting
Jan 14 16:58:30 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.44@o2ib)
Jan 14 16:58:30 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 16:58:35 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection restored to emc-OST0012 (at 192.168.1.46@o2ib)
Jan 14 16:58:35 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 14 16:58:37 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection restored to emc-OST000f (at 192.168.1.46@o2ib)
Jan 14 16:58:37 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 16:58:55 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 14 16:58:55 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 14 17:24:42 mds1 kernel: Lustre: 3315:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389738275/real 0]  req@ffff88082333f400 x1457239484387040/t0(0) o13->emc-OST0012-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389738282 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 14 17:24:42 mds1 kernel: Lustre: 3315:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 14 17:24:42 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 17:24:42 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 14 17:24:42 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 17:24:44 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 17:24:44 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 17:24:50 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389738284/real 0]  req@ffff880823307000 x1457239484387228/t0(0) o8->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389738290 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 17:24:50 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jan 14 17:25:41 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000c_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 14 17:25:41 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 14 17:25:45 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389738334/real 0]  req@ffff8808253d9000 x1457239484387824/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389738345 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 17:25:45 mds1 kernel: Lustre: 3308:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Jan 14 17:25:53 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection restored to emc-OST000f (at 192.168.1.44@o2ib)
Jan 14 17:26:32 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0013_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 14 17:26:32 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 17:27:22 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000c_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 14 17:27:22 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 17:28:12 mds1 kernel: Lustre: MGS: haven't heard from client 4790f36b-73d1-c93a-2f65-59c38654828c (at 192.168.1.46@o2ib) in 233 seconds. I think it's dead, and I am evicting it. exp ffff880822efb400, cur 1389738492 expire 1389738342 last 1389738259
Jan 14 17:28:12 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 14 17:28:12 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000c_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 14 17:28:12 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 17:32:43 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 14 17:32:43 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 17:32:43 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 14 17:32:43 mds1 kernel: LustreError: Skipped 15 previous similar messages
Jan 14 17:32:46 mds1 kernel: LustreError: 11-0: emc-OST000b-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 14 17:32:46 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 17:32:46 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 17:32:46 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 14 17:32:55 mds1 kernel: Lustre: MGS: Client 43e0376e-e8eb-da95-a4fd-4591be462d99 (at 192.168.1.46@o2ib) reconnecting
Jan 14 17:33:04 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection restored to emc-OST000b (at 192.168.1.44@o2ib)
Jan 14 17:33:04 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 14 18:37:01 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 14 18:37:01 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3146" x-info="http://www.rsyslog.com"] start
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys cpuset
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys cpu
Jan 14 18:37:01 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 14 18:37:01 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 18:37:01 mds1 kernel: KERNEL supported cpus:
Jan 14 18:37:01 mds1 kernel:  Intel GenuineIntel
Jan 14 18:37:01 mds1 kernel:  AMD AuthenticAMD
Jan 14 18:37:01 mds1 kernel:  Centaur CentaurHauls
Jan 14 18:37:01 mds1 kernel: BIOS-provided physical RAM map:
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 14 18:37:01 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 14 18:37:01 mds1 kernel: DMI 2.6 present.
Jan 14 18:37:01 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 14 18:37:01 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 14 18:37:01 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 14 18:37:01 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 14 18:37:01 mds1 kernel: Using GB pages for direct mapping
Jan 14 18:37:01 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 14 18:37:01 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 14 18:37:01 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 14 18:37:01 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 14 18:37:01 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 14 18:37:01 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 14 18:37:01 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 14 18:37:01 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 18:37:01 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 18:37:01 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 18:37:01 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 14 18:37:01 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 14 18:37:01 mds1 kernel: Setting APIC routing to flat.
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 14 18:37:01 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 14 18:37:01 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 14 18:37:01 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 14 18:37:01 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 14 18:37:01 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 14 18:37:01 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 14 18:37:01 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 14 18:37:01 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 14 18:37:01 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 14 18:37:01 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 14 18:37:01 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 14 18:37:01 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 14 18:37:01 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 14 18:37:01 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 14 18:37:01 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 14 18:37:01 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 14 18:37:01 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 14 18:37:01 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 14 18:37:01 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 14 18:37:01 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 14 18:37:01 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 14 18:37:01 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 14 18:37:01 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 14 18:37:01 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 14 18:37:01 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 14 18:37:01 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 14 18:37:01 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 14 18:37:01 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 14 18:37:01 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 14 18:37:01 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 14 18:37:01 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 14 18:37:01 mds1 kernel: Zone PFN ranges:
Jan 14 18:37:01 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 14 18:37:01 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 14 18:37:01 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 14 18:37:01 mds1 kernel: Movable zone start PFN for each node
Jan 14 18:37:01 mds1 kernel: early_node_map[5] active PFN ranges
Jan 14 18:37:01 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 14 18:37:01 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 14 18:37:01 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 14 18:37:01 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 14 18:37:01 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 14 18:37:01 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 14 18:37:01 mds1 kernel: Setting APIC routing to flat.
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 14 18:37:01 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 14 18:37:01 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 14 18:37:01 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 14 18:37:01 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 14 18:37:01 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 14 18:37:01 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 14 18:37:01 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 14 18:37:01 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 14 18:37:01 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 14 18:37:01 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 14 18:37:01 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 14 18:37:01 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 14 18:37:01 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 14 18:37:01 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 14 18:37:01 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 14 18:37:01 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 14 18:37:01 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 14 18:37:01 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 14 18:37:01 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 14 18:37:01 mds1 kernel: Policy zone: Normal
Jan 14 18:37:01 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 14 18:37:01 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 14 18:37:01 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 14 18:37:01 mds1 kernel: Checking aperture...
Jan 14 18:37:01 mds1 kernel: No AGP bridge found
Jan 14 18:37:01 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 14 18:37:01 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 14 18:37:01 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 14 18:37:01 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 14 18:37:01 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 14 18:37:01 mds1 kernel: Hierarchical RCU implementation.
Jan 14 18:37:01 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 14 18:37:01 mds1 kernel: Extended CMOS year: 2000
Jan 14 18:37:01 mds1 kernel: Console: colour VGA+ 80x25
Jan 14 18:37:01 mds1 kernel: console [tty0] enabled
Jan 14 18:37:01 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 14 18:37:01 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 14 18:37:01 mds1 kernel: Fast TSC calibration using PIT
Jan 14 18:37:01 mds1 kernel: Detected 1995.132 MHz processor.
Jan 14 18:37:01 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.26 BogoMIPS (lpj=1995132)
Jan 14 18:37:01 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 14 18:37:01 mds1 kernel: Security Framework initialized
Jan 14 18:37:01 mds1 kernel: SELinux:  Initializing.
Jan 14 18:37:01 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 14 18:37:01 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 14 18:37:01 mds1 kernel: Mount-cache hash table entries: 256
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys ns
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys memory
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys devices
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys freezer
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys net_cls
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys blkio
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys perf_event
Jan 14 18:37:01 mds1 kernel: Initializing cgroup subsys net_prio
Jan 14 18:37:01 mds1 kernel: CPU: Physical Processor ID: 0
Jan 14 18:37:01 mds1 kernel: CPU: Processor Core ID: 0
Jan 14 18:37:01 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 14 18:37:01 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 14 18:37:01 mds1 kernel: using mwait in idle threads.
Jan 14 18:37:01 mds1 kernel: ACPI: Core revision 20090903
Jan 14 18:37:01 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 14 18:37:01 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 14 18:37:01 mds1 kernel: dmar: Host address width 46
Jan 14 18:37:01 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 14 18:37:01 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 18:37:01 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 14 18:37:01 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 14 18:37:01 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 14 18:37:01 mds1 kernel: dmar: ATSR flags: 0x0
Jan 14 18:37:01 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 14 18:37:01 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 14 18:37:01 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 14 18:37:01 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 14 18:37:01 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 14 18:37:01 mds1 kernel: Enabling x2apic
Jan 14 18:37:01 mds1 kernel: Enabled x2apic
Jan 14 18:37:01 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 14 18:37:01 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 14 18:37:01 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 14 18:37:01 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 14 18:37:01 mds1 kernel: PEBS disabled due to CPU errata.
Jan 14 18:37:01 mds1 kernel: ... version:                3
Jan 14 18:37:01 mds1 kernel: ... bit width:              48
Jan 14 18:37:01 mds1 kernel: ... generic registers:      8
Jan 14 18:37:01 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 14 18:37:01 mds1 kernel: ... max period:             000000007fffffff
Jan 14 18:37:01 mds1 kernel: ... fixed-purpose events:   3
Jan 14 18:37:01 mds1 kernel: ... event mask:             00000007000000ff
Jan 14 18:37:01 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 14 18:37:01 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 14 18:37:01 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 14 18:37:01 mds1 kernel: Brought up 12 CPUs
Jan 14 18:37:01 mds1 kernel: Total of 12 processors activated (47880.64 BogoMIPS).
Jan 14 18:37:01 mds1 kernel: devtmpfs: initialized
Jan 14 18:37:01 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 14 18:37:01 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 14 18:37:01 mds1 kernel: regulator: core version 0.5
Jan 14 18:37:01 mds1 kernel: NET: Registered protocol family 16
Jan 14 18:37:01 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 14 18:37:01 mds1 kernel: ACPI: bus type pci registered
Jan 14 18:37:01 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 14 18:37:01 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 14 18:37:01 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 14 18:37:01 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 14 18:37:01 mds1 kernel: bio: create slab <bio-0> at 0
Jan 14 18:37:01 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 14 18:37:01 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 14 18:37:01 mds1 kernel: ACPI: Interpreter enabled
Jan 14 18:37:01 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 14 18:37:01 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 14 18:37:01 mds1 kernel: ACPI: No dock devices found.
Jan 14 18:37:01 mds1 kernel: HEST: Table parsing has been initialized.
Jan 14 18:37:01 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 14 18:37:01 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 14 18:37:01 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 14 18:37:01 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 14 18:37:01 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 18:37:01 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 14 18:37:01 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 14 18:37:01 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 14 18:37:01 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 14 18:37:01 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 14 18:37:01 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 18:37:01 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 14 18:37:01 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 14 18:37:01 mds1 kernel: vgaarb: loaded
Jan 14 18:37:01 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 14 18:37:01 mds1 kernel: SCSI subsystem initialized
Jan 14 18:37:01 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 14 18:37:01 mds1 kernel: usbcore: registered new interface driver hub
Jan 14 18:37:01 mds1 kernel: usbcore: registered new device driver usb
Jan 14 18:37:01 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 14 18:37:01 mds1 kernel: NetLabel: Initializing
Jan 14 18:37:01 mds1 kernel: NetLabel:  domain hash size = 128
Jan 14 18:37:01 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 14 18:37:01 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 14 18:37:01 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 14 18:37:01 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 14 18:37:01 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 14 18:37:01 mds1 kernel: Switching to clocksource hpet
Jan 14 18:37:01 mds1 kernel: pnp: PnP ACPI init
Jan 14 18:37:01 mds1 kernel: ACPI: bus type pnp registered
Jan 14 18:37:01 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 14 18:37:01 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 14 18:37:01 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 14 18:37:01 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 18:37:01 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 18:37:01 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 14 18:37:01 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 18:37:01 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 14 18:37:01 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 18:37:01 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 18:37:01 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 14 18:37:01 mds1 kernel: NET: Registered protocol family 2
Jan 14 18:37:01 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 14 18:37:01 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 14 18:37:01 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 14 18:37:01 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 14 18:37:01 mds1 kernel: TCP reno registered
Jan 14 18:37:01 mds1 kernel: NET: Registered protocol family 1
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 18:37:01 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 14 18:37:01 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 14 18:37:01 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 14 18:37:01 mds1 kernel: dmar: Unsupported device scope
Jan 14 18:37:01 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 14 18:37:01 mds1 kernel: type=2000 audit(1389742599.059:1): initialized
Jan 14 18:37:01 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 14 18:37:01 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 14 18:37:01 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 14 18:37:01 mds1 kernel: msgmni has been set to 32768
Jan 14 18:37:01 mds1 kernel: alg: No test for stdrng (krng)
Jan 14 18:37:01 mds1 kernel: ksign: Installing public key data
Jan 14 18:37:01 mds1 kernel: Loading keyring
Jan 14 18:37:01 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 14 18:37:01 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 14 18:37:01 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 14 18:37:01 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 14 18:37:01 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 14 18:37:01 mds1 kernel: io scheduler noop registered
Jan 14 18:37:01 mds1 kernel: io scheduler anticipatory registered
Jan 14 18:37:01 mds1 kernel: io scheduler deadline registered (default)
Jan 14 18:37:01 mds1 kernel: io scheduler cfq registered
Jan 14 18:37:01 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 14 18:37:01 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 14 18:37:01 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 14 18:37:01 mds1 kernel: acpiphp: Slot [2] registered
Jan 14 18:37:01 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 14 18:37:01 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 14 18:37:01 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 14 18:37:01 mds1 kernel: ipmi message handler version 39.2
Jan 14 18:37:01 mds1 kernel: IPMI System Interface driver.
Jan 14 18:37:01 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 14 18:37:01 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 14 18:37:01 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 14 18:37:01 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 14 18:37:01 mds1 kernel: IPMI kcs interface initialized
Jan 14 18:37:01 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 14 18:37:01 mds1 kernel: ACPI: Power Button [PWRF]
Jan 14 18:37:01 mds1 kernel: ERST: Could not register with persistent store
Jan 14 18:37:01 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 14 18:37:01 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 14 18:37:01 mds1 kernel: Non-volatile memory driver v1.3
Jan 14 18:37:01 mds1 kernel: Linux agpgart interface v0.103
Jan 14 18:37:01 mds1 kernel: crash memory driver: version 1.1
Jan 14 18:37:01 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 14 18:37:01 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 18:37:01 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 18:37:01 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 14 18:37:01 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 14 18:37:01 mds1 kernel: brd: module loaded
Jan 14 18:37:01 mds1 kernel: loop: module loaded
Jan 14 18:37:01 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 14 18:37:01 mds1 kernel: Fixed MDIO Bus: probed
Jan 14 18:37:01 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 14 18:37:01 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 18:37:01 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 18:37:01 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 14 18:37:01 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 18:37:01 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 14 18:37:01 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 14 18:37:01 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 14 18:37:01 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 14 18:37:01 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 14 18:37:01 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 14 18:37:01 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 14 18:37:01 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 14 18:37:01 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 14 18:37:01 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 14 18:37:01 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 14 18:37:01 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 14 18:37:01 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 14 18:37:01 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 14 18:37:01 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 14 18:37:01 mds1 kernel: Refined TSC clocksource calibration: 1995.191 MHz.
Jan 14 18:37:01 mds1 kernel: Switching to clocksource tsc
Jan 14 18:37:01 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 14 18:37:01 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 14 18:37:01 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 14 18:37:01 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 14 18:37:01 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 14 18:37:01 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 14 18:37:01 mds1 kernel: cpuidle: using governor ladder
Jan 14 18:37:01 mds1 kernel: cpuidle: using governor menu
Jan 14 18:37:01 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 14 18:37:01 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 14 18:37:01 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 14 18:37:01 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 14 18:37:01 mds1 kernel: TCP cubic registered
Jan 14 18:37:01 mds1 kernel: Initializing XFRM netlink socket
Jan 14 18:37:01 mds1 kernel: NET: Registered protocol family 17
Jan 14 18:37:01 mds1 kernel: registered taskstats version 1
Jan 14 18:37:01 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-14 23:36:41 UTC (1389742601)
Jan 14 18:37:01 mds1 kernel: Initalizing network drop monitor service
Jan 14 18:37:01 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 14 18:37:01 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 14 18:37:01 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 14 18:37:01 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 14 18:37:01 mds1 kernel: dracut: dracut-004-303.el6
Jan 14 18:37:01 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 14 18:37:01 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 14 18:37:01 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 14 18:37:01 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 14 18:37:01 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 18:37:01 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 18:37:01 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 14 18:37:01 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 14 18:37:01 mds1 kernel: udev: starting version 147
Jan 14 18:37:01 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 14 18:37:01 mds1 kernel: dracut: Starting plymouth daemon
Jan 14 18:37:01 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 14 18:37:01 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 14 18:37:01 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 14 18:37:01 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 14 18:37:01 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 14 18:37:01 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 14 18:37:01 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 14 18:37:01 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 14 18:37:01 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 14 18:37:01 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 14 18:37:01 mds1 kernel: megasas: FW now in Ready state
Jan 14 18:37:01 mds1 kernel: megasas:IOC Init cmd success
Jan 14 18:37:01 mds1 kernel: megasas: INIT adapter done
Jan 14 18:37:01 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 14 18:37:01 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 18:37:01 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 14 18:37:01 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 14 18:37:01 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 14 18:37:01 mds1 kernel: scsi1 : qla2xxx
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 14 18:37:01 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 14 18:37:01 mds1 kernel: scsi2 : qla2xxx
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 14 18:37:01 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 14 18:37:01 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 14 18:37:01 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 14 18:37:01 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 14 18:37:01 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 14 18:37:01 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 14 18:37:01 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 14 18:37:01 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 14 18:37:01 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2f00, mb[3]=4100, cmd=6a ****.
Jan 14 18:37:01 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2fa2, mb[3]=1100, cmd=6a ****.
Jan 14 18:37:01 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 14 18:37:01 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 14 18:37:01 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 14 18:37:01 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 14 18:37:01 mds1 kernel: scsi3 : ahci
Jan 14 18:37:01 mds1 kernel: scsi4 : ahci
Jan 14 18:37:01 mds1 kernel: scsi5 : ahci
Jan 14 18:37:01 mds1 kernel: scsi6 : ahci
Jan 14 18:37:01 mds1 kernel: scsi7 : ahci
Jan 14 18:37:01 mds1 kernel: scsi8 : ahci
Jan 14 18:37:01 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 14 18:37:01 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 14 18:37:01 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 14 18:37:01 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 14 18:37:01 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 14 18:37:01 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 14 18:37:01 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 14 18:37:01 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 14 18:37:01 mds1 kernel: ata1.00: configured for UDMA/100
Jan 14 18:37:01 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 14 18:37:01 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 14 18:37:01 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sdc:
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 14 18:37:01 mds1 kernel: sdb: unknown partition table
Jan 14 18:37:01 mds1 kernel: sda: unknown partition table
Jan 14 18:37:01 mds1 kernel: sdd:
Jan 14 18:37:01 mds1 kernel: sde:
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sdg: unknown partition table
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sdf: unknown partition table
Jan 14 18:37:01 mds1 kernel: unknown partition table
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: unknown partition table
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sda1 sda2 sda3
Jan 14 18:37:01 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 14 18:37:01 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 14 18:37:01 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 14 18:37:01 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 14 18:37:01 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 14 18:37:01 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 14 18:37:01 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 14 18:37:01 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 14 18:37:01 mds1 kernel: SELinux:  Disabled at runtime.
Jan 14 18:37:01 mds1 kernel: type=1404 audit(1389742604.584:2): selinux=0 auid=4294967295 ses=4294967295
Jan 14 18:37:01 mds1 kernel: dracut: 
Jan 14 18:37:01 mds1 kernel: dracut: Switching root
Jan 14 18:37:01 mds1 kernel: udev: starting version 147
Jan 14 18:37:01 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 14 18:37:01 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 14 18:37:01 mds1 kernel: PTP clock support registered
Jan 14 18:37:01 mds1 kernel: dca service started, version 1.12.1
Jan 14 18:37:01 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 14 18:37:01 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 14 18:37:01 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 14 18:37:01 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 14 18:37:01 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 14 18:37:01 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 14 18:37:01 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 14 18:37:01 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 14 18:37:01 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 14 18:37:01 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 14 18:37:01 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 14 18:37:01 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 14 18:37:01 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 14 18:37:01 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 14 18:37:01 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 14 18:37:01 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 14 18:37:01 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 14 18:37:01 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 14 18:37:01 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 14 18:37:01 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 14 18:37:01 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 14 18:37:01 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 14 18:37:01 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 14 18:37:01 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 14 18:37:01 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 14 18:37:01 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 18:37:01 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 14 18:37:01 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 18:37:01 mds1 kernel: emc: device handler registered
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 14 18:37:01 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 14 18:37:01 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 14 18:37:01 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 14 18:37:01 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 14 18:37:01 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 14 18:37:01 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 14 18:37:01 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 14 18:37:01 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028157000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 14 18:37:01 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028157000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 14 18:37:01 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 14 18:37:01 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 14 18:37:01 mds1 kernel: NET: Registered protocol family 10
Jan 14 18:37:01 mds1 kernel: lo: Disabled Privacy Extensions
Jan 14 18:37:01 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 18:37:01 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 14 18:37:01 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 14 18:37:01 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 18:37:01 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 18:37:01 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 14 18:37:01 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 14 18:37:01 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 14 18:37:01 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 14 18:37:01 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 14 18:37:01 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 14 18:37:01 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 14 18:37:01 mds1 rpc.statd[3241]: Version 1.2.3 starting
Jan 14 18:37:01 mds1 sm-notify[3242]: Version 1.2.3 starting
Jan 14 18:37:01 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 14 18:37:01 mds1 kernel: RPC: Registered udp transport module.
Jan 14 18:37:01 mds1 kernel: RPC: Registered tcp transport module.
Jan 14 18:37:01 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 14 18:37:01 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 14 18:37:01 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 14 18:37:01 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 14 18:37:01 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 14 18:37:05 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 14 18:37:05 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 14 18:37:10 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 14 18:37:10 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 14 18:37:10 mds1 attrd[3387]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:10 mds1 cib[3384]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:10 mds1 stonith-ng[3385]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:10 mds1 pengine[3388]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:11 mds1 lrmd[3386]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:11 mds1 crmd[3389]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 14 18:37:11 mds1 logger[3398]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 14 18:37:12 mds1 kdump: kexec: loaded kdump kernel
Jan 14 18:37:12 mds1 kdump: started up
Jan 14 18:37:13 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 14 18:37:13 mds1 acpid: starting up
Jan 14 18:37:13 mds1 acpid: 1 rule loaded
Jan 14 18:37:13 mds1 acpid: waiting for events: event logging is off
Jan 14 18:37:13 mds1 acpid: client connected from 3601[68:68]
Jan 14 18:37:13 mds1 acpid: 1 client rule loaded
Jan 14 18:37:21 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 14 18:37:23 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 14 18:37:33 mds1 kernel: ipmi device interface
Jan 14 18:37:35 mds1 automount[3770]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 14 18:37:35 mds1 abrtd: Init complete, entering main loop
Jan 14 18:37:36 mds1 cimserver[3928]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 14 18:37:40 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 18:37:40 mds1 kernel: 
Jan 14 18:37:40 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 18:37:40 mds1 kernel: 
Jan 14 18:37:40 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 14 18:37:40 mds1 kernel: 
Jan 14 18:38:22 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 14 18:38:23 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 14 18:38:23 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 14 18:38:23 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 18:38:23 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 18:38:23 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 18:38:28 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389742703/real 0]  req@ffff880fd6c40c00 x1457250767994888/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389742708 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:38:28 mds1 kernel: LustreError: 4164:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 14 18:38:28 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 18:38:28 mds1 kernel: LustreError: 4387:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 18:38:28 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 18:38:28 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 14 18:38:28 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 18:38:28 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 14 18:38:28 mds1 kernel: LustreError: 4456:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 14 18:38:28 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 14 18:38:33 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389742708/real 0]  req@ffff880fcfd43400 x1457250767995340/t0(0) o8->emc-OST0005-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389742713 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:38:47 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 18:38:53 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -11.
Jan 14 18:38:59 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389742733/real 0]  req@ffff880fd25d0c00 x1457250767995684/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389742739 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:38:59 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jan 14 18:39:18 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 18:39:29 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389742758/real 0]  req@ffff880fd25d0400 x1457250767995944/t0(0) o8->emc-OST0005-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389742769 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:40:44 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 14 18:40:44 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 14 18:40:44 mds1 kernel: LustreError: 5056:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 14 18:40:44 mds1 kernel: Lustre: 4509:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 14 18:40:44 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 1:57, of 5 clients 0 recovered and 0 were evicted.
Jan 14 18:40:44 mds1 kernel: LustreError: 5058:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0000: Aborting recovery
Jan 14 18:40:44 mds1 kernel: Lustre: 4455:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 14 18:40:44 mds1 kernel: Lustre: 4455:0:(ldlm_lib.c:1798:target_recovery_overseer()) Skipped 2 previous similar messages
Jan 14 18:40:44 mds1 kernel: LustreError: 4455:0:(ldlm_lib.c:1465:abort_lock_replay_queue()) @@@ aborted:  req@ffff88082c958c00 x1457244015225800/t0(0) o101->74f7c566-80b6-fb5a-f6c9-826abdbb58b9@192.168.1.57@o2ib:0/0 lens 328/0 e 1 to 0 dl 1389742860 ref 1 fl Complete:/40/ffffffff rc 0/-1
Jan 14 18:40:44 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 1:26, of 5 clients 0 recovered and 0 were evicted.
Jan 14 18:40:44 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 192.168.1.57@o2ib (stopping)
Jan 14 18:40:44 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 0@lo (stopping)
Jan 14 18:40:44 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 14 18:40:44 mds1 kernel: LustreError: Skipped 15 previous similar messages
Jan 14 18:40:44 mds1 kernel: LustreError: 4428:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST000d-osc-MDT0000: cannot cleanup orphans: rc = -5
Jan 14 18:40:44 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 14 18:40:54 mds1 kernel: Lustre: 5054:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389742844/real 1389742844]  req@ffff88082825bc00 x1457250767997020/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389742854 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:40:54 mds1 kernel: Lustre: 5054:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jan 14 18:40:54 mds1 kernel: Lustre: server umount MGS complete
Jan 14 18:40:54 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 14 18:41:45 mds1 cibadmin[5173]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 18:41:45 mds1 cibadmin[5174]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 18:41:47 mds1 cibadmin[5178]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 18:41:47 mds1 cibadmin[5179]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 18:41:48 mds1 cibadmin[5183]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 14 18:41:48 mds1 cibadmin[5184]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 14 18:42:36 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 18:42:38 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 14 18:42:38 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 14 18:42:38 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 14 18:42:38 mds1 kernel: LustreError: Skipped 23 previous similar messages
Jan 14 18:42:38 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389742958/real 1389742958]  req@ffff880803d20400 x1457250767997124/t0(0) o8->emc-OST0007-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389742963 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 14 18:42:38 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 14 18:42:38 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 14 18:43:03 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 14 18:43:03 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 14 18:43:03 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 14 18:43:09 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 14 18:43:28 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 14 18:43:28 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 14 18:43:53 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 14 18:43:53 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 14 18:46:40 mds1 kernel: INFO: task tgt_recov:5585 blocked for more than 120 seconds.
Jan 14 18:46:40 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 18:46:40 mds1 kernel: tgt_recov     D 0000000000000001     0  5585      2 0x00000080
Jan 14 18:46:40 mds1 kernel: ffff880803661da0 0000000000000046 0000000000000000 ffff880045a16740
Jan 14 18:46:40 mds1 kernel: ffff880803661d30 ffffffff81055f96 ffff880803661d40 ffff8808281af500
Jan 14 18:46:40 mds1 kernel: ffff880803659058 ffff880803661fd8 000000000000fb88 ffff880803659058
Jan 14 18:46:40 mds1 kernel: Call Trace:
Jan 14 18:46:40 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 14 18:46:40 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 14 18:46:40 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 14 18:46:40 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 14 18:46:40 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 14 18:46:40 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 14 18:47:05 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 14 18:48:09 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 14 18:48:09 mds1 kernel: Lustre: emc-MDT0000: disconnecting 2 stale clients
Jan 14 18:49:09 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 14 18:49:09 mds1 kernel: Lustre: emc-MDT0000: disconnecting 1 stale clients
Jan 14 18:49:09 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 6:00, of 5 clients 2 recovered and 3 were evicted.
Jan 14 18:50:33 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 3cb8395f-703e-cad9-a803-c38db0f89d44 (at 192.168.1.56@o2ib) in 433 seconds. I think it's dead, and I am evicting it. exp ffff8808043ca800, cur 1389743433 expire 1389743283 last 1389743000
Jan 14 18:55:34 mds1 kernel: LustreError: 11-0: emc-OST0015-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 14 18:55:34 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 18:55:34 mds1 kernel: LustreError: Skipped 19 previous similar messages
Jan 14 18:55:34 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 18:55:34 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 14 18:55:46 mds1 kernel: Lustre: MGS: Client 7c11bd17-4e5b-2f31-ad63-fd2a7e44743d (at 192.168.1.44@o2ib) reconnecting
Jan 14 18:55:53 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection restored to emc-OST000b (at 192.168.1.44@o2ib)
Jan 14 18:55:58 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 14 18:55:58 mds1 kernel: LustreError: Skipped 20 previous similar messages
Jan 14 18:55:58 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection to emc-OST0017 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 14 18:55:58 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 14 18:56:08 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 14 18:56:08 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 14 19:28:01 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x6d:0x0], rc = 0 [1]
Jan 14 20:34:56 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 1761c3bc-877b-851d-1750-319123916f70 (at 192.168.1.55@o2ib) in 232 seconds. I think it's dead, and I am evicting it. exp ffff88082a4a2400, cur 1389749696 expire 1389749546 last 1389749464
Jan 15 07:41:53 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:41:59 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:05 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:11 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:17 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:23 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:35 mds1 kernel: LustreError: 0-0: emc-MDT0000: trigger OI scrub by RPC for [0x8000090a2:0x10b:0x0], rc = 0 [1]
Jan 15 07:42:35 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 14:23:23 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 15 14:23:23 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3161" x-info="http://www.rsyslog.com"] start
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys cpuset
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys cpu
Jan 15 14:23:23 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 15 14:23:23 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 15 14:23:23 mds1 kernel: KERNEL supported cpus:
Jan 15 14:23:23 mds1 kernel:  Intel GenuineIntel
Jan 15 14:23:23 mds1 kernel:  AMD AuthenticAMD
Jan 15 14:23:23 mds1 kernel:  Centaur CentaurHauls
Jan 15 14:23:23 mds1 kernel: BIOS-provided physical RAM map:
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 15 14:23:23 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 15 14:23:23 mds1 kernel: DMI 2.6 present.
Jan 15 14:23:23 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 15 14:23:23 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 15 14:23:23 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 15 14:23:23 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 15 14:23:23 mds1 kernel: Using GB pages for direct mapping
Jan 15 14:23:23 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 15 14:23:23 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 15 14:23:23 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 15 14:23:23 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 15 14:23:23 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 15 14:23:23 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 15 14:23:23 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 14:23:23 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 14:23:23 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 14:23:23 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 14:23:23 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 14:23:23 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 15 14:23:23 mds1 kernel: Setting APIC routing to flat.
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 15 14:23:23 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 15 14:23:23 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 15 14:23:23 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 15 14:23:23 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 15 14:23:23 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 15 14:23:23 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 15 14:23:23 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 15 14:23:23 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 15 14:23:23 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 15 14:23:23 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 15 14:23:23 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 15 14:23:23 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 15 14:23:23 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 15 14:23:23 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 15 14:23:23 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 15 14:23:23 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 15 14:23:23 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 15 14:23:23 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 15 14:23:23 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 15 14:23:23 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 15 14:23:23 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 15 14:23:23 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 15 14:23:23 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 15 14:23:23 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 15 14:23:23 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 15 14:23:23 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 15 14:23:23 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 15 14:23:23 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 15 14:23:23 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 15 14:23:23 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 15 14:23:23 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 15 14:23:23 mds1 kernel: Zone PFN ranges:
Jan 15 14:23:23 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 15 14:23:23 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 15 14:23:23 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 15 14:23:23 mds1 kernel: Movable zone start PFN for each node
Jan 15 14:23:23 mds1 kernel: early_node_map[5] active PFN ranges
Jan 15 14:23:23 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 15 14:23:23 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 15 14:23:23 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 15 14:23:23 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 15 14:23:23 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 15 14:23:23 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 15 14:23:23 mds1 kernel: Setting APIC routing to flat.
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 15 14:23:23 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 15 14:23:23 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 15 14:23:23 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 15 14:23:23 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 15 14:23:23 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 15 14:23:23 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 15 14:23:23 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 15 14:23:23 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 15 14:23:23 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 15 14:23:23 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 15 14:23:23 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 15 14:23:23 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 15 14:23:23 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 15 14:23:23 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 15 14:23:23 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 15 14:23:23 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 15 14:23:23 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 15 14:23:23 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 15 14:23:23 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 15 14:23:23 mds1 kernel: Policy zone: Normal
Jan 15 14:23:23 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 15 14:23:23 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 15 14:23:23 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 15 14:23:23 mds1 kernel: Checking aperture...
Jan 15 14:23:23 mds1 kernel: No AGP bridge found
Jan 15 14:23:23 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 15 14:23:23 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 15 14:23:23 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 15 14:23:23 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 15 14:23:23 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 15 14:23:23 mds1 kernel: Hierarchical RCU implementation.
Jan 15 14:23:23 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 15 14:23:23 mds1 kernel: Extended CMOS year: 2000
Jan 15 14:23:23 mds1 kernel: Console: colour VGA+ 80x25
Jan 15 14:23:23 mds1 kernel: console [tty0] enabled
Jan 15 14:23:23 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 15 14:23:23 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 15 14:23:23 mds1 kernel: Fast TSC calibration using PIT
Jan 15 14:23:23 mds1 kernel: Detected 1995.142 MHz processor.
Jan 15 14:23:23 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.28 BogoMIPS (lpj=1995142)
Jan 15 14:23:23 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 15 14:23:23 mds1 kernel: Security Framework initialized
Jan 15 14:23:23 mds1 kernel: SELinux:  Initializing.
Jan 15 14:23:23 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 15 14:23:23 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 15 14:23:23 mds1 kernel: Mount-cache hash table entries: 256
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys ns
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys memory
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys devices
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys freezer
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys net_cls
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys blkio
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys perf_event
Jan 15 14:23:23 mds1 kernel: Initializing cgroup subsys net_prio
Jan 15 14:23:23 mds1 kernel: CPU: Physical Processor ID: 0
Jan 15 14:23:23 mds1 kernel: CPU: Processor Core ID: 0
Jan 15 14:23:23 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 15 14:23:23 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 15 14:23:23 mds1 kernel: using mwait in idle threads.
Jan 15 14:23:23 mds1 kernel: ACPI: Core revision 20090903
Jan 15 14:23:23 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 15 14:23:23 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 15 14:23:23 mds1 kernel: dmar: Host address width 46
Jan 15 14:23:23 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 15 14:23:23 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 15 14:23:23 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 15 14:23:23 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 15 14:23:23 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 15 14:23:23 mds1 kernel: dmar: ATSR flags: 0x0
Jan 15 14:23:23 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 15 14:23:23 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 15 14:23:23 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 15 14:23:23 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 15 14:23:23 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 15 14:23:23 mds1 kernel: Enabling x2apic
Jan 15 14:23:23 mds1 kernel: Enabled x2apic
Jan 15 14:23:23 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 15 14:23:23 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 15 14:23:23 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 15 14:23:23 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 15 14:23:23 mds1 kernel: PEBS disabled due to CPU errata.
Jan 15 14:23:23 mds1 kernel: ... version:                3
Jan 15 14:23:23 mds1 kernel: ... bit width:              48
Jan 15 14:23:23 mds1 kernel: ... generic registers:      8
Jan 15 14:23:23 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 15 14:23:23 mds1 kernel: ... max period:             000000007fffffff
Jan 15 14:23:23 mds1 kernel: ... fixed-purpose events:   3
Jan 15 14:23:23 mds1 kernel: ... event mask:             00000007000000ff
Jan 15 14:23:23 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 15 14:23:23 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 15 14:23:23 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 15 14:23:23 mds1 kernel: Brought up 12 CPUs
Jan 15 14:23:23 mds1 kernel: Total of 12 processors activated (47880.78 BogoMIPS).
Jan 15 14:23:23 mds1 kernel: devtmpfs: initialized
Jan 15 14:23:23 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 15 14:23:23 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 15 14:23:23 mds1 kernel: regulator: core version 0.5
Jan 15 14:23:23 mds1 kernel: NET: Registered protocol family 16
Jan 15 14:23:23 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 15 14:23:23 mds1 kernel: ACPI: bus type pci registered
Jan 15 14:23:23 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 15 14:23:23 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 15 14:23:23 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 15 14:23:23 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 15 14:23:23 mds1 kernel: bio: create slab <bio-0> at 0
Jan 15 14:23:23 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 15 14:23:23 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 15 14:23:23 mds1 kernel: ACPI: Interpreter enabled
Jan 15 14:23:23 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 15 14:23:23 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 15 14:23:23 mds1 kernel: ACPI: No dock devices found.
Jan 15 14:23:23 mds1 kernel: HEST: Table parsing has been initialized.
Jan 15 14:23:23 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 15 14:23:23 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 15 14:23:23 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 15 14:23:23 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 15 14:23:23 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 14:23:23 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 15 14:23:23 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 15 14:23:23 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 15 14:23:23 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 15 14:23:23 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 15 14:23:23 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 14:23:23 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 14:23:23 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 15 14:23:23 mds1 kernel: vgaarb: loaded
Jan 15 14:23:23 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 15 14:23:23 mds1 kernel: SCSI subsystem initialized
Jan 15 14:23:23 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 15 14:23:23 mds1 kernel: usbcore: registered new interface driver hub
Jan 15 14:23:23 mds1 kernel: usbcore: registered new device driver usb
Jan 15 14:23:23 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 15 14:23:23 mds1 kernel: NetLabel: Initializing
Jan 15 14:23:23 mds1 kernel: NetLabel:  domain hash size = 128
Jan 15 14:23:23 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 15 14:23:23 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 15 14:23:23 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 15 14:23:23 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 15 14:23:23 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 15 14:23:23 mds1 kernel: Switching to clocksource hpet
Jan 15 14:23:23 mds1 kernel: pnp: PnP ACPI init
Jan 15 14:23:23 mds1 kernel: ACPI: bus type pnp registered
Jan 15 14:23:23 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 15 14:23:23 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 15 14:23:23 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 15 14:23:23 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 14:23:23 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 14:23:23 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 14:23:23 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 14:23:23 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 15 14:23:23 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 14:23:23 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 14:23:23 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 14:23:23 mds1 kernel: NET: Registered protocol family 2
Jan 15 14:23:23 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 15 14:23:23 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 15 14:23:23 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 15 14:23:23 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 15 14:23:23 mds1 kernel: TCP reno registered
Jan 15 14:23:23 mds1 kernel: NET: Registered protocol family 1
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 15 14:23:23 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 15 14:23:23 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 15 14:23:23 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 15 14:23:23 mds1 kernel: dmar: Unsupported device scope
Jan 15 14:23:23 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 15 14:23:23 mds1 kernel: type=2000 audit(1389813781.056:1): initialized
Jan 15 14:23:23 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 15 14:23:23 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 15 14:23:23 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 15 14:23:23 mds1 kernel: msgmni has been set to 32768
Jan 15 14:23:23 mds1 kernel: alg: No test for stdrng (krng)
Jan 15 14:23:23 mds1 kernel: ksign: Installing public key data
Jan 15 14:23:23 mds1 kernel: Loading keyring
Jan 15 14:23:23 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 15 14:23:23 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 15 14:23:23 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 15 14:23:23 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 15 14:23:23 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 15 14:23:23 mds1 kernel: io scheduler noop registered
Jan 15 14:23:23 mds1 kernel: io scheduler anticipatory registered
Jan 15 14:23:23 mds1 kernel: io scheduler deadline registered (default)
Jan 15 14:23:23 mds1 kernel: io scheduler cfq registered
Jan 15 14:23:23 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 15 14:23:23 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 15 14:23:23 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 15 14:23:23 mds1 kernel: acpiphp: Slot [2] registered
Jan 15 14:23:23 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 15 14:23:23 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 15 14:23:23 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 15 14:23:23 mds1 kernel: ipmi message handler version 39.2
Jan 15 14:23:23 mds1 kernel: IPMI System Interface driver.
Jan 15 14:23:23 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 15 14:23:23 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 15 14:23:23 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 15 14:23:23 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 15 14:23:23 mds1 kernel: IPMI kcs interface initialized
Jan 15 14:23:23 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 15 14:23:23 mds1 kernel: ACPI: Power Button [PWRF]
Jan 15 14:23:23 mds1 kernel: ERST: Could not register with persistent store
Jan 15 14:23:23 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 15 14:23:23 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 15 14:23:23 mds1 kernel: Non-volatile memory driver v1.3
Jan 15 14:23:23 mds1 kernel: Linux agpgart interface v0.103
Jan 15 14:23:23 mds1 kernel: crash memory driver: version 1.1
Jan 15 14:23:23 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 15 14:23:23 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 15 14:23:23 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 15 14:23:23 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 15 14:23:23 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 15 14:23:23 mds1 kernel: brd: module loaded
Jan 15 14:23:23 mds1 kernel: loop: module loaded
Jan 15 14:23:23 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 15 14:23:23 mds1 kernel: Fixed MDIO Bus: probed
Jan 15 14:23:23 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 15 14:23:23 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 15 14:23:23 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 15 14:23:23 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 15 14:23:23 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 15 14:23:23 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 15 14:23:23 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 15 14:23:23 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 15 14:23:23 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 15 14:23:23 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 15 14:23:23 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 15 14:23:23 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 15 14:23:23 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 15 14:23:23 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 15 14:23:23 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 15 14:23:23 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 15 14:23:23 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 15 14:23:23 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 15 14:23:23 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 15 14:23:23 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 15 14:23:23 mds1 kernel: Refined TSC clocksource calibration: 1995.191 MHz.
Jan 15 14:23:23 mds1 kernel: Switching to clocksource tsc
Jan 15 14:23:23 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 15 14:23:23 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 15 14:23:23 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 15 14:23:23 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 15 14:23:23 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 15 14:23:23 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 15 14:23:23 mds1 kernel: cpuidle: using governor ladder
Jan 15 14:23:23 mds1 kernel: cpuidle: using governor menu
Jan 15 14:23:23 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 15 14:23:23 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 15 14:23:23 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 15 14:23:23 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 15 14:23:23 mds1 kernel: TCP cubic registered
Jan 15 14:23:23 mds1 kernel: Initializing XFRM netlink socket
Jan 15 14:23:23 mds1 kernel: NET: Registered protocol family 17
Jan 15 14:23:23 mds1 kernel: registered taskstats version 1
Jan 15 14:23:23 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-15 19:23:03 UTC (1389813783)
Jan 15 14:23:23 mds1 kernel: Initalizing network drop monitor service
Jan 15 14:23:23 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 15 14:23:23 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 15 14:23:23 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 15 14:23:23 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 15 14:23:23 mds1 kernel: dracut: dracut-004-303.el6
Jan 15 14:23:23 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 15 14:23:23 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 15 14:23:23 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 15 14:23:23 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 15 14:23:23 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 15 14:23:23 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 15 14:23:23 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 15 14:23:23 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 15 14:23:23 mds1 kernel: udev: starting version 147
Jan 15 14:23:23 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 15 14:23:23 mds1 kernel: dracut: Starting plymouth daemon
Jan 15 14:23:23 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 15 14:23:23 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 15 14:23:23 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 15 14:23:23 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 15 14:23:23 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 15 14:23:23 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 15 14:23:23 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 15 14:23:23 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 15 14:23:23 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 15 14:23:23 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 15 14:23:23 mds1 kernel: megasas: FW now in Ready state
Jan 15 14:23:23 mds1 kernel: megasas:IOC Init cmd success
Jan 15 14:23:23 mds1 kernel: megasas: INIT adapter done
Jan 15 14:23:23 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 15 14:23:23 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 15 14:23:23 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 15 14:23:23 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 15 14:23:23 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 15 14:23:23 mds1 kernel: scsi1 : qla2xxx
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 15 14:23:23 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 15 14:23:23 mds1 kernel: scsi2 : qla2xxx
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 15 14:23:23 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 15 14:23:23 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 15 14:23:23 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 15 14:23:23 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 15 14:23:23 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 15 14:23:23 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 15 14:23:23 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 15 14:23:23 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 15 14:23:23 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dde, mb[3]=100, cmd=6a ****.
Jan 15 14:23:23 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dfc, mb[3]=4100, cmd=6a ****.
Jan 15 14:23:23 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 14:23:23 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 15 14:23:23 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 15 14:23:23 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 15 14:23:23 mds1 kernel: scsi3 : ahci
Jan 15 14:23:23 mds1 kernel: scsi4 : ahci
Jan 15 14:23:23 mds1 kernel: scsi5 : ahci
Jan 15 14:23:23 mds1 kernel: scsi6 : ahci
Jan 15 14:23:23 mds1 kernel: scsi7 : ahci
Jan 15 14:23:23 mds1 kernel: scsi8 : ahci
Jan 15 14:23:23 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 15 14:23:23 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 15 14:23:23 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 15 14:23:23 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 15 14:23:23 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 15 14:23:23 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 15 14:23:23 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 15 14:23:23 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 15 14:23:23 mds1 kernel: ata1.00: configured for UDMA/100
Jan 15 14:23:23 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 15 14:23:23 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 15 14:23:23 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sda:
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 14:23:23 mds1 kernel: sde:
Jan 15 14:23:23 mds1 kernel: sdd:
Jan 15 14:23:23 mds1 kernel: sdb:
Jan 15 14:23:23 mds1 kernel: sdc:
Jan 15 14:23:23 mds1 kernel: sdf: unknown partition table
Jan 15 14:23:23 mds1 kernel: unknown partition table
Jan 15 14:23:23 mds1 kernel: unknown partition table
Jan 15 14:23:23 mds1 kernel: sdg: unknown partition table
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: unknown partition table
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: unknown partition table
Jan 15 14:23:23 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: sda1 sda2 sda3
Jan 15 14:23:23 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 15 14:23:23 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 15 14:23:23 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 15 14:23:23 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 15 14:23:23 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 15 14:23:23 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 15 14:23:23 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 15 14:23:23 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 15 14:23:23 mds1 kernel: SELinux:  Disabled at runtime.
Jan 15 14:23:23 mds1 kernel: type=1404 audit(1389813787.344:2): selinux=0 auid=4294967295 ses=4294967295
Jan 15 14:23:23 mds1 kernel: dracut: 
Jan 15 14:23:23 mds1 kernel: dracut: Switching root
Jan 15 14:23:23 mds1 kernel: udev: starting version 147
Jan 15 14:23:23 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 15 14:23:23 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 15 14:23:23 mds1 kernel: PTP clock support registered
Jan 15 14:23:23 mds1 kernel: dca service started, version 1.12.1
Jan 15 14:23:23 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 15 14:23:23 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 15 14:23:23 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 14:23:24 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 15 14:23:24 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 15 14:23:24 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 15 14:23:24 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 15 14:23:24 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 15 14:23:24 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 15 14:23:24 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 15 14:23:24 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 15 14:23:24 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 15 14:23:24 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 15 14:23:24 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 15 14:23:24 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 14:23:24 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 15 14:23:24 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 15 14:23:24 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 15 14:23:24 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 14:23:24 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 14:23:24 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 15 14:23:24 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 15 14:23:24 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 15 14:23:24 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 15 14:23:24 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 15 14:23:24 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 15 14:23:24 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 14:23:24 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 15 14:23:24 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 14:23:24 mds1 kernel: emc: device handler registered
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 15 14:23:24 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 15 14:23:24 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 15 14:23:24 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 15 14:23:24 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 15 14:23:24 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 15 14:23:24 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 15 14:23:24 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 15 14:23:24 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10294a5000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 15 14:23:24 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10294a5000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 15 14:23:24 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 15 14:23:24 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 15 14:23:24 mds1 kernel: NET: Registered protocol family 10
Jan 15 14:23:24 mds1 kernel: lo: Disabled Privacy Extensions
Jan 15 14:23:24 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 15 14:23:24 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 15 14:23:24 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 15 14:23:24 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 15 14:23:24 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 14:23:24 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 15 14:23:24 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 14:23:24 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 15 14:23:24 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 15 14:23:24 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 15 14:23:24 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 15 14:23:24 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 15 14:23:24 mds1 rpc.statd[3256]: Version 1.2.3 starting
Jan 15 14:23:24 mds1 sm-notify[3257]: Version 1.2.3 starting
Jan 15 14:23:24 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 15 14:23:24 mds1 kernel: RPC: Registered udp transport module.
Jan 15 14:23:24 mds1 kernel: RPC: Registered tcp transport module.
Jan 15 14:23:24 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 15 14:23:24 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 15 14:23:24 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 15 14:23:24 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 15 14:23:24 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 15 14:23:28 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 15 14:23:28 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 15 14:23:32 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 15 14:23:33 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 15 14:23:33 mds1 cib[3399]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:33 mds1 attrd[3402]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:33 mds1 pengine[3403]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:33 mds1 stonith-ng[3400]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:33 mds1 lrmd[3401]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:33 mds1 crmd[3404]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 14:23:34 mds1 logger[3413]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 15 14:23:35 mds1 kdump: kexec: loaded kdump kernel
Jan 15 14:23:35 mds1 kdump: started up
Jan 15 14:23:38 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 15 14:23:38 mds1 acpid: starting up
Jan 15 14:23:38 mds1 acpid: 1 rule loaded
Jan 15 14:23:38 mds1 acpid: waiting for events: event logging is off
Jan 15 14:23:38 mds1 acpid: client connected from 3619[68:68]
Jan 15 14:23:38 mds1 acpid: 1 client rule loaded
Jan 15 14:23:38 mds1 kernel: ipmi device interface
Jan 15 14:23:39 mds1 automount[3658]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 15 14:23:40 mds1 abrtd: Init complete, entering main loop
Jan 15 14:23:41 mds1 cimserver[3809]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 15 14:23:44 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 15 14:23:46 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 15 14:24:10 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 14:24:16 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389813851/real 1389813851]  req@ffff880814905c00 x1457325408780292/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389813856 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 14:24:22 mds1 kernel: LustreError: 4191:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880814905800 x1457325408780296/t0(0) o253->MGC192.168.1.40@o2ib@0@lo:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:24:22 mds1 kernel: LustreError: 4191:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 15 14:24:22 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 14:24:22 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 15 14:24:22 mds1 kernel: LustreError: 4371:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 15 14:24:27 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389813862/real 1389813862]  req@ffff881008a1fc00 x1457325408780312/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389813867 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 14:24:28 mds1 kernel: LustreError: 4191:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880802760c00 x1457325408780508/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:24:34 mds1 kernel: LustreError: 4245:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880814905400 x1457325408780512/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:24:34 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 14:24:40 mds1 kernel: LustreError: 4245:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880814905400 x1457325408780520/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:24:40 mds1 kernel: LustreError: 4245:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 15 14:24:40 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 15 14:24:46 mds1 kernel: LustreError: 4191:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880802760c00 x1457325408780532/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:24:46 mds1 kernel: LustreError: 4191:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jan 15 14:24:46 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 14:24:57 mds1 kernel: LustreError: 4191:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880802760c00 x1457325408780536/t0(0) o253->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:25:05 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 14:25:05 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 15 14:25:08 mds1 kernel: LustreError: 4347:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880802760c00 x1457325408780540/t0(0) o253->MGC192.168.1.40@o2ib@0@lo:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:25:08 mds1 kernel: LustreError: 4347:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 15 14:25:14 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 15 14:25:14 mds1 kernel: LustreError: 4485:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 15 14:25:14 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 15 14:25:15 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389813905/real 1389813905]  req@ffff88082fb2a000 x1457325408780544/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389813915 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 14:25:15 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 15 14:25:26 mds1 kernel: LustreError: 4347:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8807ff84a000 x1457325408780856/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 14:25:26 mds1 kernel: LustreError: 4347:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Jan 15 14:25:26 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 14:25:28 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 5 clients reconnect
Jan 15 14:25:38 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 14:25:42 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389813932/real 1389813932]  req@ffff880832fd6c00 x1457325408780868/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389813942 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 14:25:50 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 15 14:25:50 mds1 kernel: LustreError: 4724:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 15 14:25:50 mds1 kernel: Lustre: 4572:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 15 14:25:50 mds1 kernel: LustreError: 4501:0:(osp_precreate.c:736:osp_precreate_cleanup_orphans()) emc-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5
Jan 15 14:25:50 mds1 kernel: LustreError: 4503:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0001-osc-MDT0001: can't precreate: rc = -5
Jan 15 14:25:50 mds1 kernel: LustreError: 4505:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0002-osc-MDT0001: cannot precreate objects: rc = -5
Jan 15 14:25:50 mds1 kernel: LustreError: 4503:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 1 previous similar message
Jan 15 14:25:50 mds1 kernel: LustreError: 4505:0:(osp_precreate.c:986:osp_precreate_thread()) Skipped 3 previous similar messages
Jan 15 14:25:50 mds1 kernel: LustreError: 4511:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0005-osc-MDT0001: can't precreate: rc = -5
Jan 15 14:25:50 mds1 kernel: LustreError: 4489:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0006-osc-MDT0001: cannot precreate objects: rc = -5
Jan 15 14:25:50 mds1 kernel: LustreError: 4511:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 5 previous similar messages
Jan 15 14:25:51 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 14:25:51 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 15 14:25:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.57@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 14:25:54 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 14:25:57 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 14:25:57 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 14:25:57 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 15 14:26:22 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 15 14:26:42 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 1:14, of 5 clients 5 recovered and 0 were evicted.
Jan 15 16:41:49 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 16:41:49 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 15 16:41:49 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 16:41:49 mds1 kernel: Lustre: emc-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 15 16:41:55 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 2:30, or until 5 clients reconnect
Jan 15 16:42:21 mds1 kernel: Lustre: 3377:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389822134/real 1389822134]  req@ffff88080ac48800 x1457325409017788/t0(0) o400->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1389822141 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 16:42:21 mds1 kernel: Lustre: 3377:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 15 16:42:21 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 16:42:27 mds1 kernel: Lustre: 3372:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389822141/real 1389822141]  req@ffff88080b15e400 x1457325409017996/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389822147 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 16:42:46 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:51, of 5 clients 5 recovered and 0 were evicted.
Jan 15 16:42:46 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 15 17:33:36 mds1 kernel: Lustre: 3373:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389824291/real 1389824291]  req@ffff880812010800 x1457325409109476/t0(0) o6->emc-OST0007-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 664/432 e 5 to 1 dl 1389825216 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 17:33:36 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection to emc-OST0007 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 17:33:36 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:34:01 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:34:26 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:34:51 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:35:16 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:35:41 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:36:06 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:36:31 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:36:53 mds1 kernel: Lustre: 3383:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389824290/real 1389824290]  req@ffff880812537800 x1457325409108640/t0(0) o6->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 664/432 e 4 to 1 dl 1389825412 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 17:36:53 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 17:36:54 mds1 kernel: Lustre: 3373:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389824293/real 1389824293]  req@ffff880812237400 x1457325409110472/t0(0) o6->emc-OST0007-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 664/432 e 4 to 1 dl 1389825414 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 17:36:54 mds1 kernel: Lustre: 3373:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 15 17:36:55 mds1 kernel: Lustre: 3378:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389824292/real 1389824292]  req@ffff88080a7b5c00 x1457325409110220/t0(0) o6->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 664/432 e 4 to 1 dl 1389825415 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 17:36:55 mds1 kernel: Lustre: 3378:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jan 15 17:37:18 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:37:18 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 17:38:33 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:38:33 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 15 17:41:03 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:41:03 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 15 17:45:38 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:45:38 mds1 kernel: LustreError: Skipped 21 previous similar messages
Jan 15 17:54:23 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 17:54:23 mds1 kernel: LustreError: Skipped 41 previous similar messages
Jan 15 18:04:48 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:04:48 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 18:15:13 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:15:13 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 18:25:38 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:25:38 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 18:36:03 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:36:03 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 18:46:28 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:46:28 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 18:56:53 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 18:56:53 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:07:18 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:07:18 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:17:43 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:17:43 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:28:08 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:28:08 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:28:24 mds1 kernel: LustreError: 138-a: emc-MDT0000: A client on nid 192.168.1.55@o2ib was evicted due to a lock blocking callback time out: rc -107
Jan 15 19:30:56 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 9d9fb6c5-1582-e41a-33b0-1871f4f0cb8c (at 192.168.1.55@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff880812d72000, cur 1389832256 expire 1389832106 last 1389832028
Jan 15 19:36:58 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:38:33 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:38:33 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:38:38 mds1 kernel: LNet: Service thread pid 4832 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 19:38:38 mds1 kernel: Pid: 4832, comm: mdt02_004
Jan 15 19:38:38 mds1 kernel: 
Jan 15 19:38:38 mds1 kernel: Call Trace:
Jan 15 19:38:38 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 19:38:38 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:38:38 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 19:38:38 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0ee4fdc>] osp_precreate_timeout_condition+0xcc/0xe0 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81055ad3>] ? __wake_up+0x53/0x70
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0ee35c2>] osp_object_free+0x182/0x190 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0ee4b75>] ? osp_declare_object_create+0x155/0x4f0 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2f2dd>] ? lod_qos_declare_object_on+0xed/0x480 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e30ddf>] ? lod_alloc_rr.clone.2+0x66f/0xde0 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e325f4>] ? lod_qos_prep_create+0x10a4/0x1aa0 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa08fddfa>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0901b32>] ? fld_server_lookup+0x72/0x430 [fld]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2bb2b>] ? lod_declare_striped_object+0x14b/0x880 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2c721>] ? lod_declare_object_create+0x4c1/0x790 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e878ef>] ? mdd_declare_object_create_internal+0xbf/0x1f0 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffff811a58f7>] ? generic_getxattr+0x87/0x90
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e96eae>] ? mdd_declare_create+0x4e/0x870 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e953cf>] ? mdd_linkea_prepare+0x24f/0x4e0 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e97e91>] ? mdd_create+0x7c1/0x1730 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0c99787>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e28560>] ? lod_index_lookup+0x0/0x30 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d67dc8>] ? mdo_create+0x18/0x50 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d72031>] ? mdt_reint_open+0x1351/0x20a0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0432e16>] ? upcall_cache_get_entry+0x296/0x880 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0597600>] ? lu_ucred_global_init+0x0/0x30 [obdclass]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d5aeb1>] ? mdt_reint_rec+0x41/0xe0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d42c93>] ? mdt_reint_internal+0x4c3/0x780 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d4321d>] ? mdt_intent_reint+0x1ed/0x520 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d3e8ce>] ? mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa06b1461>] ? ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa06da17f>] ? ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d3ed96>] ? mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d45a8a>] ? mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d7fc55>] ? mds_regular_handle+0x15/0x20 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0709e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 19:38:38 mds1 kernel: [<ffffffffa070b18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 15 19:38:38 mds1 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 15 19:38:38 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 19:38:38 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 19:38:38 mds1 kernel: 
Jan 15 19:38:38 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389832718.4832
Jan 15 19:38:38 mds1 kernel: LNet: Service thread pid 4250 was inactive for 200.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 19:38:38 mds1 kernel: Pid: 4250, comm: mdt00_002
Jan 15 19:38:38 mds1 kernel: 
Jan 15 19:38:38 mds1 kernel: Call Trace:
Jan 15 19:38:38 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 15 19:38:38 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 19:38:38 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0eeb14c>] osp_precreate_reserve+0x5dc/0x1ef0 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0ee35c2>] ? osp_object_free+0x182/0x190 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0ee4b75>] osp_declare_object_create+0x155/0x4f0 [osp]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2f2dd>] lod_qos_declare_object_on+0xed/0x480 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e30ddf>] lod_alloc_rr.clone.2+0x66f/0xde0 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e325f4>] lod_qos_prep_create+0x10a4/0x1aa0 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa08fddfa>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0901b32>] ? fld_server_lookup+0x72/0x430 [fld]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2bb2b>] lod_declare_striped_object+0x14b/0x880 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e2c721>] lod_declare_object_create+0x4c1/0x790 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e878ef>] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffff811a58f7>] ? generic_getxattr+0x87/0x90
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e96eae>] mdd_declare_create+0x4e/0x870 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e953cf>] ? mdd_linkea_prepare+0x24f/0x4e0 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e97e91>] mdd_create+0x7c1/0x1730 [mdd]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0c99787>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0e28560>] ? lod_index_lookup+0x0/0x30 [lod]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d67dc8>] mdo_create+0x18/0x50 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d72031>] mdt_reint_open+0x1351/0x20a0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0432e16>] ? upcall_cache_get_entry+0x296/0x880 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0597600>] ? lu_ucred_global_init+0x0/0x30 [obdclass]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d5aeb1>] mdt_reint_rec+0x41/0xe0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d42c93>] mdt_reint_internal+0x4c3/0x780 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d4321d>] mdt_intent_reint+0x1ed/0x520 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 19:38:38 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 19:38:38 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 15 19:38:38 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 15 19:38:38 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 19:38:38 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 19:38:38 mds1 kernel: 
Jan 15 19:38:49 mds1 kernel: LNet: Service thread pid 4888 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 19:38:49 mds1 kernel: Pid: 4888, comm: mdt00_003
Jan 15 19:38:49 mds1 kernel: 
Jan 15 19:38:49 mds1 kernel: Call Trace:
Jan 15 19:38:49 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 19:38:49 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06cc8a0>] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d10b1>] ldlm_completion_ast+0x4b1/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0810>] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d40ccb>] mdt_object_lock0+0x33b/0xaf0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d41544>] mdt_object_lock+0x14/0x20 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d4f359>] mdt_getattr_name_lock+0xe19/0x1990 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06f8f85>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa071ff06>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06fb214>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d50169>] mdt_intent_getattr+0x299/0x480 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 19:38:49 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 15 19:38:49 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 15 19:38:49 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 19:38:49 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 19:38:49 mds1 kernel: 
Jan 15 19:38:49 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389832729.4888
Jan 15 19:38:49 mds1 kernel: Pid: 4249, comm: mdt00_001
Jan 15 19:38:49 mds1 kernel: 
Jan 15 19:38:49 mds1 kernel: Call Trace:
Jan 15 19:38:49 mds1 kernel: [<ffffffff811b59d7>] ? __find_get_block+0x97/0xe0
Jan 15 19:38:49 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 19:38:49 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06cc8a0>] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d10b1>] ldlm_completion_ast+0x4b1/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0810>] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d40ccb>] mdt_object_lock0+0x33b/0xaf0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d41544>] mdt_object_lock+0x14/0x20 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d4f359>] mdt_getattr_name_lock+0xe19/0x1990 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06f8f85>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa071ff06>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06fb214>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d50169>] mdt_intent_getattr+0x299/0x480 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 19:38:49 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 19:38:49 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 15 19:38:49 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 15 19:38:49 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 19:38:49 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 19:38:49 mds1 kernel: 
Jan 15 19:40:18 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:40:18 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 19:40:29 mds1 kernel: Lustre: lock timed out (enqueued at 1389832529, 300s ago)
Jan 15 19:40:29 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 19:40:29 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389832829.4888
Jan 15 19:40:43 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 67e6c951-2969-39c7-6196-503f3b0cb043 (at 192.168.1.55@o2ib) in 237 seconds. I think it's dead, and I am evicting it. exp ffff88080b67dc00, cur 1389832843 expire 1389832693 last 1389832606
Jan 15 19:40:52 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 67e6c951-2969-39c7-6196-503f3b0cb043 (at 192.168.1.55@o2ib) in 246 seconds. I think it's dead, and I am evicting it. exp ffff88080bf2d800, cur 1389832852 expire 1389832702 last 1389832606
Jan 15 19:41:58 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:41:58 mds1 kernel: LustreError: 4250:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88080b67dc00 ns: mdt-emc-MDT0000_UUID lock: ffff88080b87e900/0xcd96244e36165e83 lrc: 3/0,0 mode: CR/CR res: [0x800009870:0x1a:0x0].0 bits 0x9 rrc: 1 type: IBT flags: 0x50200000000000 nid: 192.168.1.55@o2ib remote: 0xb93be1bcbec3c3d9 expref: 7 pid: 4250 timeout: 0 lvb_type: 0
Jan 15 19:41:58 mds1 kernel: LNet: Service thread pid 4249 completed after 389.19s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jan 15 19:41:58 mds1 kernel: LustreError: 4250:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue0()) Skipped 2 previous similar messages
Jan 15 19:41:58 mds1 kernel: Lustre: 4250:0:(service.c:2030:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (396:4s); client may timeout.  req@ffff880812553400 x1457258333558832/t536870916723(0) o101->67e6c951-2969-39c7-6196-503f3b0cb043@192.168.1.55@o2ib:0/0 lens 592/1608 e 4 to 0 dl 1389832914 ref 1 fl Complete:/0/0 rc -107/-107
Jan 15 19:43:38 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:43:38 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 19:45:18 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:46:58 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:48:38 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:48:58 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:48:58 mds1 kernel: LustreError: Skipped 49 previous similar messages
Jan 15 19:50:08 mds1 kernel: Lustre: 3382:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389832486/real 1389832486]  req@ffff880812239400 x1457325409426712/t0(0) o6->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 664/432 e 5 to 1 dl 1389833408 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 19:50:08 mds1 kernel: Lustre: 3382:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 15 19:50:08 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 19:50:18 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:51:58 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:55:18 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4121:0x0], next=[0x100070000:0x4121:0x0], reserved=0, syn_changes=46, syn_rpc_in_progress=6, status=-19
Jan 15 19:55:18 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 19:59:18 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client ec809b6e-fa04-4123-c3e7-c0c9ba30064a (at 192.168.1.54@o2ib) in 256 seconds. I think it's dead, and I am evicting it. exp ffff88080ab25000, cur 1389833958 expire 1389833808 last 1389833702
Jan 15 19:59:18 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 19:59:18 mds1 kernel: LustreError: Skipped 69 previous similar messages
Jan 15 20:00:18 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x40e1:0x0], next=[0x100060000:0x40e1:0x0], reserved=0, syn_changes=58, syn_rpc_in_progress=6, status=-19
Jan 15 20:00:18 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 20:01:58 mds1 kernel: LustreError: 4255:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue0()) ### lock on destroyed export ffff881008890800 ns: mdt-emc-MDT0000_UUID lock: ffff880ff6a34280/0xcd96244e361660e4 lrc: 3/0,0 mode: CR/CR res: [0x8000090a2:0x33b:0x0].0 bits 0x9 rrc: 1 type: IBT flags: 0x50200000000000 nid: 192.168.1.54@o2ib remote: 0xaebdb5669be1a0f1 expref: 3 pid: 4255 timeout: 0 lvb_type: 0
Jan 15 20:09:43 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:09:43 mds1 kernel: LustreError: Skipped 74 previous similar messages
Jan 15 20:10:46 mds1 init: tty (/dev/tty2) main process (3836) killed by TERM signal
Jan 15 20:10:46 mds1 init: tty (/dev/tty3) main process (3838) killed by TERM signal
Jan 15 20:10:46 mds1 init: tty (/dev/tty4) main process (3840) killed by TERM signal
Jan 15 20:10:46 mds1 init: tty (/dev/tty5) main process (3842) killed by TERM signal
Jan 15 20:10:46 mds1 init: tty (/dev/tty6) main process (3844) killed by TERM signal
Jan 15 20:10:46 mds1 abrtd: Got signal 15, exiting
Jan 15 20:10:50 mds1 acpid: exiting
Jan 15 20:10:50 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 15 20:10:50 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 15 20:10:51 mds1 kernel: Lustre: server umount MGS complete
Jan 15 20:10:56 mds1 kernel: Lustre: 39282:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834650/real 1389834650]  req@ffff8808043b0c00 x1457325409509256/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389834656 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:10:56 mds1 kernel: Lustre: 39282:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jan 15 20:10:56 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 15 20:10:56 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:10:57 mds1 multipathd: mgt: stop event checker thread (140456840705792)
Jan 15 20:10:57 mds1 multipathd: mdt2: stop event checker thread (140456829961984)
Jan 15 20:10:57 mds1 multipathd: mdt1: stop event checker thread (140456829929216)
Jan 15 20:10:57 mds1 multipathd: --------shut down-------
Jan 15 20:10:57 mds1 init: Disconnected from system bus
Jan 15 20:10:57 mds1 console-kit-daemon[3848]: WARNING: no sender#012
Jan 15 20:10:57 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 15 20:10:57 mds1 auditd[3136]: The audit daemon is exiting.
Jan 15 20:10:57 mds1 kernel: type=1305 audit(1389834657.761:594): audit_pid=0 old=3136 auid=4294967295 ses=4294967295 res=1
Jan 15 20:10:57 mds1 kernel: type=1305 audit(1389834657.856:595): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Jan 15 20:10:57 mds1 kernel: Kernel logging (proc) stopped.
Jan 15 20:10:57 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3161" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 15 20:13:44 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 15 20:13:44 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3100" x-info="http://www.rsyslog.com"] start
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys cpuset
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys cpu
Jan 15 20:13:44 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 15 20:13:44 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 15 20:13:44 mds1 kernel: KERNEL supported cpus:
Jan 15 20:13:44 mds1 kernel:  Intel GenuineIntel
Jan 15 20:13:44 mds1 kernel:  AMD AuthenticAMD
Jan 15 20:13:44 mds1 kernel:  Centaur CentaurHauls
Jan 15 20:13:44 mds1 kernel: BIOS-provided physical RAM map:
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 15 20:13:44 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 15 20:13:44 mds1 kernel: DMI 2.6 present.
Jan 15 20:13:44 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 15 20:13:44 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 15 20:13:44 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 15 20:13:44 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 15 20:13:44 mds1 kernel: Using GB pages for direct mapping
Jan 15 20:13:44 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 15 20:13:44 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 15 20:13:44 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 15 20:13:44 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 15 20:13:44 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 15 20:13:44 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 15 20:13:44 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 15 20:13:44 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 20:13:44 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 20:13:44 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 20:13:44 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 15 20:13:44 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 15 20:13:44 mds1 kernel: Setting APIC routing to flat.
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 15 20:13:44 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 15 20:13:44 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 15 20:13:44 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 15 20:13:44 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 15 20:13:44 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 15 20:13:44 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 15 20:13:44 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 15 20:13:44 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 15 20:13:44 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 15 20:13:44 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 15 20:13:44 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 15 20:13:44 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 15 20:13:44 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 15 20:13:44 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 15 20:13:44 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 15 20:13:44 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 15 20:13:44 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 15 20:13:44 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 15 20:13:44 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 15 20:13:44 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 15 20:13:44 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 15 20:13:44 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 15 20:13:44 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 15 20:13:44 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 15 20:13:44 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 15 20:13:44 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 15 20:13:44 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 15 20:13:44 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 15 20:13:44 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 15 20:13:44 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 15 20:13:44 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 15 20:13:44 mds1 kernel: Zone PFN ranges:
Jan 15 20:13:44 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 15 20:13:44 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 15 20:13:44 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 15 20:13:44 mds1 kernel: Movable zone start PFN for each node
Jan 15 20:13:44 mds1 kernel: early_node_map[5] active PFN ranges
Jan 15 20:13:44 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 15 20:13:44 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 15 20:13:44 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 15 20:13:44 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 15 20:13:44 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 15 20:13:44 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 15 20:13:44 mds1 kernel: Setting APIC routing to flat.
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 15 20:13:44 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 15 20:13:44 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 15 20:13:44 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 15 20:13:44 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 15 20:13:44 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 15 20:13:44 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 15 20:13:44 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 15 20:13:44 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 15 20:13:44 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 15 20:13:44 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 15 20:13:44 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 15 20:13:44 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 15 20:13:44 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 15 20:13:44 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 15 20:13:44 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 15 20:13:44 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 15 20:13:44 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 15 20:13:44 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 15 20:13:44 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 15 20:13:44 mds1 kernel: Policy zone: Normal
Jan 15 20:13:44 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 15 20:13:44 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 15 20:13:44 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 15 20:13:44 mds1 kernel: Checking aperture...
Jan 15 20:13:44 mds1 kernel: No AGP bridge found
Jan 15 20:13:44 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 15 20:13:44 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 15 20:13:44 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 15 20:13:44 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 15 20:13:44 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 15 20:13:44 mds1 kernel: Hierarchical RCU implementation.
Jan 15 20:13:44 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 15 20:13:44 mds1 kernel: Extended CMOS year: 2000
Jan 15 20:13:44 mds1 kernel: Console: colour VGA+ 80x25
Jan 15 20:13:44 mds1 kernel: console [tty0] enabled
Jan 15 20:13:44 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 15 20:13:44 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 15 20:13:44 mds1 kernel: Fast TSC calibration using PIT
Jan 15 20:13:44 mds1 kernel: Detected 1995.117 MHz processor.
Jan 15 20:13:44 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.23 BogoMIPS (lpj=1995117)
Jan 15 20:13:44 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 15 20:13:44 mds1 kernel: Security Framework initialized
Jan 15 20:13:44 mds1 kernel: SELinux:  Initializing.
Jan 15 20:13:44 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 15 20:13:44 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 15 20:13:44 mds1 kernel: Mount-cache hash table entries: 256
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys ns
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys memory
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys devices
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys freezer
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys net_cls
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys blkio
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys perf_event
Jan 15 20:13:44 mds1 kernel: Initializing cgroup subsys net_prio
Jan 15 20:13:44 mds1 kernel: CPU: Physical Processor ID: 0
Jan 15 20:13:44 mds1 kernel: CPU: Processor Core ID: 0
Jan 15 20:13:44 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 15 20:13:44 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 15 20:13:44 mds1 kernel: using mwait in idle threads.
Jan 15 20:13:44 mds1 kernel: ACPI: Core revision 20090903
Jan 15 20:13:44 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 15 20:13:44 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 15 20:13:44 mds1 kernel: dmar: Host address width 46
Jan 15 20:13:44 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 15 20:13:44 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 15 20:13:44 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 15 20:13:44 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 15 20:13:44 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 15 20:13:44 mds1 kernel: dmar: ATSR flags: 0x0
Jan 15 20:13:44 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 15 20:13:44 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 15 20:13:44 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 15 20:13:44 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 15 20:13:44 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 15 20:13:44 mds1 kernel: Enabling x2apic
Jan 15 20:13:44 mds1 kernel: Enabled x2apic
Jan 15 20:13:44 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 15 20:13:44 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 15 20:13:44 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 15 20:13:44 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 15 20:13:44 mds1 kernel: PEBS disabled due to CPU errata.
Jan 15 20:13:44 mds1 kernel: ... version:                3
Jan 15 20:13:44 mds1 kernel: ... bit width:              48
Jan 15 20:13:44 mds1 kernel: ... generic registers:      8
Jan 15 20:13:44 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 15 20:13:44 mds1 kernel: ... max period:             000000007fffffff
Jan 15 20:13:44 mds1 kernel: ... fixed-purpose events:   3
Jan 15 20:13:44 mds1 kernel: ... event mask:             00000007000000ff
Jan 15 20:13:44 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 15 20:13:44 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 15 20:13:44 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 15 20:13:44 mds1 kernel: Brought up 12 CPUs
Jan 15 20:13:44 mds1 kernel: Total of 12 processors activated (47880.48 BogoMIPS).
Jan 15 20:13:44 mds1 kernel: devtmpfs: initialized
Jan 15 20:13:44 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 15 20:13:44 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 15 20:13:44 mds1 kernel: regulator: core version 0.5
Jan 15 20:13:44 mds1 kernel: NET: Registered protocol family 16
Jan 15 20:13:44 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 15 20:13:44 mds1 kernel: ACPI: bus type pci registered
Jan 15 20:13:44 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 15 20:13:44 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 15 20:13:44 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 15 20:13:44 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 15 20:13:44 mds1 kernel: bio: create slab <bio-0> at 0
Jan 15 20:13:44 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 15 20:13:44 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 15 20:13:44 mds1 kernel: ACPI: Interpreter enabled
Jan 15 20:13:44 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 15 20:13:44 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 15 20:13:44 mds1 kernel: ACPI: No dock devices found.
Jan 15 20:13:44 mds1 kernel: HEST: Table parsing has been initialized.
Jan 15 20:13:44 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 15 20:13:44 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 15 20:13:44 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 15 20:13:44 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 15 20:13:44 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 20:13:44 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 15 20:13:44 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 15 20:13:44 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 15 20:13:44 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 15 20:13:44 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 15 20:13:44 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 20:13:44 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 15 20:13:44 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 15 20:13:44 mds1 kernel: vgaarb: loaded
Jan 15 20:13:44 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 15 20:13:44 mds1 kernel: SCSI subsystem initialized
Jan 15 20:13:44 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 15 20:13:44 mds1 kernel: usbcore: registered new interface driver hub
Jan 15 20:13:44 mds1 kernel: usbcore: registered new device driver usb
Jan 15 20:13:44 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 15 20:13:44 mds1 kernel: NetLabel: Initializing
Jan 15 20:13:44 mds1 kernel: NetLabel:  domain hash size = 128
Jan 15 20:13:44 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 15 20:13:44 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 15 20:13:44 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 15 20:13:44 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 15 20:13:44 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 15 20:13:44 mds1 kernel: Switching to clocksource hpet
Jan 15 20:13:44 mds1 kernel: pnp: PnP ACPI init
Jan 15 20:13:44 mds1 kernel: ACPI: bus type pnp registered
Jan 15 20:13:44 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 15 20:13:44 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 15 20:13:44 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 15 20:13:44 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 20:13:44 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 20:13:44 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 15 20:13:44 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 20:13:44 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 15 20:13:44 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 20:13:44 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 20:13:44 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 15 20:13:44 mds1 kernel: NET: Registered protocol family 2
Jan 15 20:13:44 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 15 20:13:44 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 15 20:13:44 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 15 20:13:44 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 15 20:13:44 mds1 kernel: TCP reno registered
Jan 15 20:13:44 mds1 kernel: NET: Registered protocol family 1
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 15 20:13:44 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 15 20:13:44 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 15 20:13:44 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 15 20:13:44 mds1 kernel: dmar: Unsupported device scope
Jan 15 20:13:44 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 15 20:13:44 mds1 kernel: type=2000 audit(1389834803.055:1): initialized
Jan 15 20:13:44 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 15 20:13:44 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 15 20:13:44 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 15 20:13:44 mds1 kernel: msgmni has been set to 32768
Jan 15 20:13:44 mds1 kernel: alg: No test for stdrng (krng)
Jan 15 20:13:44 mds1 kernel: ksign: Installing public key data
Jan 15 20:13:44 mds1 kernel: Loading keyring
Jan 15 20:13:44 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 15 20:13:44 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 15 20:13:44 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 15 20:13:44 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 15 20:13:44 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 15 20:13:44 mds1 kernel: io scheduler noop registered
Jan 15 20:13:44 mds1 kernel: io scheduler anticipatory registered
Jan 15 20:13:44 mds1 kernel: io scheduler deadline registered (default)
Jan 15 20:13:44 mds1 kernel: io scheduler cfq registered
Jan 15 20:13:44 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 15 20:13:44 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 15 20:13:44 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 15 20:13:44 mds1 kernel: acpiphp: Slot [2] registered
Jan 15 20:13:44 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 15 20:13:44 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 15 20:13:44 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 15 20:13:44 mds1 kernel: ipmi message handler version 39.2
Jan 15 20:13:44 mds1 kernel: IPMI System Interface driver.
Jan 15 20:13:44 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 15 20:13:44 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 15 20:13:44 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 15 20:13:44 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 15 20:13:44 mds1 kernel: IPMI kcs interface initialized
Jan 15 20:13:44 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 15 20:13:44 mds1 kernel: ACPI: Power Button [PWRF]
Jan 15 20:13:44 mds1 kernel: ERST: Could not register with persistent store
Jan 15 20:13:44 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 15 20:13:44 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 15 20:13:44 mds1 kernel: Non-volatile memory driver v1.3
Jan 15 20:13:44 mds1 kernel: Linux agpgart interface v0.103
Jan 15 20:13:44 mds1 kernel: crash memory driver: version 1.1
Jan 15 20:13:44 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 15 20:13:44 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 15 20:13:44 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 15 20:13:44 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 15 20:13:44 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 15 20:13:44 mds1 kernel: brd: module loaded
Jan 15 20:13:44 mds1 kernel: loop: module loaded
Jan 15 20:13:44 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 15 20:13:44 mds1 kernel: Fixed MDIO Bus: probed
Jan 15 20:13:44 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 15 20:13:44 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 15 20:13:44 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 15 20:13:44 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 15 20:13:44 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 15 20:13:44 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 15 20:13:44 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 15 20:13:44 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 15 20:13:44 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 15 20:13:44 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 15 20:13:44 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 15 20:13:44 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 15 20:13:44 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 15 20:13:44 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 15 20:13:44 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 15 20:13:44 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 15 20:13:44 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 15 20:13:44 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 15 20:13:44 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 15 20:13:44 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 15 20:13:44 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 15 20:13:44 mds1 kernel: Switching to clocksource tsc
Jan 15 20:13:44 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 15 20:13:44 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 15 20:13:44 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 15 20:13:44 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 15 20:13:44 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 15 20:13:44 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 15 20:13:44 mds1 kernel: cpuidle: using governor ladder
Jan 15 20:13:44 mds1 kernel: cpuidle: using governor menu
Jan 15 20:13:44 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 15 20:13:44 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 15 20:13:44 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 15 20:13:44 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 15 20:13:44 mds1 kernel: TCP cubic registered
Jan 15 20:13:44 mds1 kernel: Initializing XFRM netlink socket
Jan 15 20:13:44 mds1 kernel: NET: Registered protocol family 17
Jan 15 20:13:44 mds1 kernel: registered taskstats version 1
Jan 15 20:13:44 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-16 01:13:25 UTC (1389834805)
Jan 15 20:13:44 mds1 kernel: Initalizing network drop monitor service
Jan 15 20:13:44 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 15 20:13:44 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 15 20:13:44 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 15 20:13:44 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 15 20:13:44 mds1 kernel: dracut: dracut-004-303.el6
Jan 15 20:13:44 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 15 20:13:44 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 15 20:13:44 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 15 20:13:44 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 15 20:13:44 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 15 20:13:44 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 15 20:13:44 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 15 20:13:44 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 15 20:13:44 mds1 kernel: udev: starting version 147
Jan 15 20:13:44 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 15 20:13:44 mds1 kernel: dracut: Starting plymouth daemon
Jan 15 20:13:44 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 15 20:13:44 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 15 20:13:44 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 15 20:13:44 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 15 20:13:44 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 15 20:13:44 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 15 20:13:44 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 15 20:13:44 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 15 20:13:44 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 15 20:13:44 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 15 20:13:44 mds1 kernel: megasas: FW now in Ready state
Jan 15 20:13:44 mds1 kernel: megasas:IOC Init cmd success
Jan 15 20:13:44 mds1 kernel: megasas: INIT adapter done
Jan 15 20:13:44 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 15 20:13:44 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 15 20:13:44 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 15 20:13:44 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 15 20:13:44 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 15 20:13:44 mds1 kernel: scsi1 : qla2xxx
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 15 20:13:44 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 15 20:13:44 mds1 kernel: scsi2 : qla2xxx
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 15 20:13:44 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 15 20:13:44 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 15 20:13:44 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 15 20:13:44 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 15 20:13:44 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 15 20:13:44 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 15 20:13:44 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 15 20:13:44 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 15 20:13:44 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2d62, mb[3]=b100, cmd=6a ****.
Jan 15 20:13:44 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2d01, mb[3]=d100, cmd=6a ****.
Jan 15 20:13:44 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 15 20:13:44 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 15 20:13:44 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 15 20:13:44 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 15 20:13:44 mds1 kernel: scsi3 : ahci
Jan 15 20:13:44 mds1 kernel: scsi4 : ahci
Jan 15 20:13:44 mds1 kernel: scsi5 : ahci
Jan 15 20:13:44 mds1 kernel: scsi6 : ahci
Jan 15 20:13:44 mds1 kernel: scsi7 : ahci
Jan 15 20:13:44 mds1 kernel: scsi8 : ahci
Jan 15 20:13:44 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 15 20:13:44 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 15 20:13:44 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 15 20:13:44 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 15 20:13:44 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 15 20:13:44 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 15 20:13:44 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 15 20:13:44 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 15 20:13:44 mds1 kernel: ata1.00: configured for UDMA/100
Jan 15 20:13:44 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 15 20:13:44 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 15 20:13:44 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: sda:
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 15 20:13:44 mds1 kernel: sdc:
Jan 15 20:13:44 mds1 kernel: sdb:
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 15 20:13:44 mds1 kernel: unknown partition table
Jan 15 20:13:44 mds1 kernel: sdd: unknown partition table
Jan 15 20:13:44 mds1 kernel: sdf:
Jan 15 20:13:44 mds1 kernel: sde:
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: unknown partition table
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sdg: unknown partition table
Jan 15 20:13:44 mds1 kernel: unknown partition table
Jan 15 20:13:44 mds1 kernel: unknown partition table
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sda1 sda2 sda3
Jan 15 20:13:44 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 15 20:13:44 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 15 20:13:44 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 15 20:13:44 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 15 20:13:44 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 15 20:13:44 mds1 kernel: SELinux:  Disabled at runtime.
Jan 15 20:13:44 mds1 kernel: type=1404 audit(1389834808.251:2): selinux=0 auid=4294967295 ses=4294967295
Jan 15 20:13:44 mds1 kernel: dracut: 
Jan 15 20:13:44 mds1 kernel: dracut: Switching root
Jan 15 20:13:44 mds1 kernel: readahead: starting
Jan 15 20:13:44 mds1 kernel: udev: starting version 147
Jan 15 20:13:44 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 15 20:13:44 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 15 20:13:44 mds1 kernel: PTP clock support registered
Jan 15 20:13:44 mds1 kernel: dca service started, version 1.12.1
Jan 15 20:13:44 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 15 20:13:44 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 15 20:13:44 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 15 20:13:44 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 15 20:13:44 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 15 20:13:44 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 15 20:13:44 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 15 20:13:44 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 15 20:13:44 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 15 20:13:44 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 15 20:13:44 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 15 20:13:44 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 15 20:13:44 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 15 20:13:44 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 15 20:13:44 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 15 20:13:44 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 15 20:13:44 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 15 20:13:44 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 15 20:13:44 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 15 20:13:44 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 15 20:13:44 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 15 20:13:44 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 15 20:13:44 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 15 20:13:44 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 15 20:13:44 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 15 20:13:44 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 20:13:44 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 15 20:13:44 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 20:13:44 mds1 kernel: emc: device handler registered
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 15 20:13:44 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 15 20:13:44 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 15 20:13:44 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 15 20:13:44 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 15 20:13:44 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 15 20:13:44 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 15 20:13:44 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 15 20:13:44 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102904b000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 15 20:13:44 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x102904b000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 15 20:13:44 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 15 20:13:44 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 15 20:13:44 mds1 kernel: NET: Registered protocol family 10
Jan 15 20:13:44 mds1 kernel: lo: Disabled Privacy Extensions
Jan 15 20:13:44 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 15 20:13:44 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 15 20:13:44 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 15 20:13:44 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 15 20:13:44 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 20:13:44 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 15 20:13:44 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 15 20:13:44 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 15 20:13:44 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 15 20:13:44 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 15 20:13:44 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 15 20:13:45 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 15 20:13:45 mds1 rpc.statd[3195]: Version 1.2.3 starting
Jan 15 20:13:45 mds1 sm-notify[3196]: Version 1.2.3 starting
Jan 15 20:13:45 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 15 20:13:45 mds1 kernel: RPC: Registered udp transport module.
Jan 15 20:13:45 mds1 kernel: RPC: Registered tcp transport module.
Jan 15 20:13:45 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 15 20:13:45 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 15 20:13:45 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 15 20:13:45 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 15 20:13:45 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 15 20:13:49 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 15 20:13:49 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 15 20:13:53 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 15 20:13:53 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 15 20:13:54 mds1 cib[3338]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:54 mds1 attrd[3341]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:54 mds1 crmd[3343]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:54 mds1 pengine[3342]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:54 mds1 stonith-ng[3339]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:54 mds1 lrmd[3340]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 15 20:13:55 mds1 logger[3352]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 15 20:13:56 mds1 kdump: kexec: loaded kdump kernel
Jan 15 20:13:56 mds1 kdump: started up
Jan 15 20:13:58 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 15 20:13:58 mds1 acpid: starting up
Jan 15 20:13:58 mds1 acpid: 1 rule loaded
Jan 15 20:13:58 mds1 acpid: waiting for events: event logging is off
Jan 15 20:13:59 mds1 acpid: client connected from 3560[68:68]
Jan 15 20:13:59 mds1 acpid: 1 client rule loaded
Jan 15 20:13:59 mds1 kernel: ipmi device interface
Jan 15 20:14:00 mds1 automount[3599]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 15 20:14:01 mds1 abrtd: Init complete, entering main loop
Jan 15 20:14:01 mds1 cimserver[3750]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 15 20:14:06 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 15 20:14:08 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 15 20:14:09 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 20:14:11 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.42@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 20:14:15 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834850/real 1389834850]  req@ffff880ffc187c00 x1457347449847812/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389834855 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:14:21 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ffc187800 x1457347449847816/t0(0) o253->MGC192.168.1.40@o2ib@0@lo:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:21 mds1 kernel: LustreError: 4006:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0001: error registering with the MGS: rc = -5 (not fatal)
Jan 15 20:14:27 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ffc187800 x1457347449847820/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:33 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ffc187800 x1457347449847824/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:33 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 15 20:14:33 mds1 kernel: LustreError: 4140:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 15 20:14:33 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 20:14:33 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 15 20:14:39 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff8355400 x1457347449848024/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:45 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff8355400 x1457347449848028/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:45 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 20:14:51 mds1 kernel: LustreError: 4006:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880ff8355400 x1457347449848032/t0(0) o101->MGC192.168.1.40@o2ib@0@lo:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:14:51 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 20:14:51 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 15 20:14:51 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 20:14:59 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 20:14:59 mds1 kernel: LNet: 3303:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 192.168.1.42@o2ib
Jan 15 20:15:05 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 15 20:15:05 mds1 kernel: LustreError: 166-1: MGC192.168.1.40@o2ib: Connection to MGS (at 192.168.1.42@o2ib) was lost; in progress operations using this service will fail
Jan 15 20:15:05 mds1 kernel: LustreError: 4326:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 15 20:15:05 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 15 20:15:05 mds1 kernel: LustreError: 4434:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 15 20:15:10 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834905/real 1389834905]  req@ffff880ff8355000 x1457347449848088/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389834910 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:15:11 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834905/real 1389834905]  req@ffff88082f857000 x1457347449848092/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389834911 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:15:11 mds1 kernel: LustreError: 4326:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88082fa1f400 x1457347449848452/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:15:17 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 20:15:23 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 15 20:15:29 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834923/real 1389834923]  req@ffff88082fa1f400 x1457347449848480/t0(0) o38->emc-MDT0000-lwp-MDT0001@192.168.1.42@o2ib:12/10 lens 400/544 e 0 to 1 dl 1389834929 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:15:29 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 15 20:15:29 mds1 kernel: LustreError: 4326:0:(client.c:1080:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880815cd4400 x1457347449848484/t0(0) o101->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jan 15 20:15:29 mds1 kernel: LustreError: 4326:0:(client.c:1080:ptlrpc_import_delay_req()) Skipped 5 previous similar messages
Jan 15 20:15:29 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 15 20:15:48 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 1 client reconnects
Jan 15 20:15:48 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Jan 15 20:15:48 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 15 20:28:22 mds1 kernel: Lustre: 3323:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834905/real 1389834905]  req@ffff88082fa1f800 x1457347449848224/t0(0) o6->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 664/432 e 1 to 1 dl 1389835702 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 20:28:22 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:28:22 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection to emc-OST0007 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:28:22 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:28:47 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:28:47 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 20:29:12 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:29:12 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 20:29:37 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:29:37 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 20:30:02 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:30:02 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 20:30:27 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:30:27 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 15 20:30:28 mds1 kernel: Lustre: 3323:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834905/real 1389834905]  req@ffff880832f94800 x1457347449848372/t0(0) o6->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 664/432 e 5 to 1 dl 1389835828 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 20:30:28 mds1 kernel: Lustre: 3323:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 15 20:30:28 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:30:29 mds1 kernel: Lustre: 3312:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389834905/real 1389834905]  req@ffff880832f94000 x1457347449848380/t0(0) o6->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 664/432 e 5 to 1 dl 1389835828 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 20:30:29 mds1 kernel: Lustre: 3312:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 15 20:30:53 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 15 20:30:53 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 15 20:31:18 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:31:18 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 15 20:31:59 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:32:08 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:32:08 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 15 20:33:23 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 15 20:33:23 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 15 20:33:39 mds1 kernel: LNet: Service thread pid 4069 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 20:33:39 mds1 kernel: Pid: 4069, comm: mdt02_000
Jan 15 20:33:39 mds1 kernel: 
Jan 15 20:33:39 mds1 kernel: Call Trace:
Jan 15 20:33:39 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:33:39 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:33:39 mds1 kernel: [<ffffffff810810cc>] ? lock_timer_base+0x3c/0x70
Jan 15 20:33:39 mds1 kernel: [<ffffffff8105231d>] ? check_preempt_curr+0x6d/0x90
Jan 15 20:33:39 mds1 kernel: [<ffffffff8106327e>] ? try_to_wake_up+0x24e/0x3e0
Jan 15 20:33:39 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 15 20:33:39 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0eeb14c>] ? osp_precreate_reserve+0x5dc/0x1ef0 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0ee35c2>] ? osp_object_free+0x182/0x190 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0ee4b75>] ? osp_declare_object_create+0x155/0x4f0 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2f2dd>] ? lod_qos_declare_object_on+0xed/0x480 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e30ddf>] ? lod_alloc_rr.clone.2+0x66f/0xde0 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e325f4>] ? lod_qos_prep_create+0x10a4/0x1aa0 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa08fddfa>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0901b32>] ? fld_server_lookup+0x72/0x430 [fld]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2bb2b>] ? lod_declare_striped_object+0x14b/0x880 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2c721>] ? lod_declare_object_create+0x4c1/0x790 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e878ef>] ? mdd_declare_object_create_internal+0xbf/0x1f0 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffff811a58f7>] ? generic_getxattr+0x87/0x90
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e96eae>] ? mdd_declare_create+0x4e/0x870 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e953cf>] ? mdd_linkea_prepare+0x24f/0x4e0 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e97e91>] ? mdd_create+0x7c1/0x1730 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0c99787>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e28560>] ? lod_index_lookup+0x0/0x30 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d67dc8>] ? mdo_create+0x18/0x50 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d72031>] ? mdt_reint_open+0x1351/0x20a0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0432e16>] ? upcall_cache_get_entry+0x296/0x880 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0597600>] ? lu_ucred_global_init+0x0/0x30 [obdclass]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d5aeb1>] ? mdt_reint_rec+0x41/0xe0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d42c93>] ? mdt_reint_internal+0x4c3/0x780 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d4321d>] ? mdt_intent_reint+0x1ed/0x520 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d3e8ce>] ? mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa06b1461>] ? ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa06da17f>] ? ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d3ed96>] ? mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d45a8a>] ? mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d7fc55>] ? mds_regular_handle+0x15/0x20 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0709e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 20:33:39 mds1 kernel: [<ffffffffa070b18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan 15 20:33:39 mds1 kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan 15 20:33:39 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 20:33:39 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 20:33:39 mds1 kernel: 
Jan 15 20:33:39 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389836019.4069
Jan 15 20:33:39 mds1 kernel: LNet: Service thread pid 5646 was inactive for 200.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 20:33:39 mds1 kernel: Pid: 5646, comm: mdt00_005
Jan 15 20:33:39 mds1 kernel: 
Jan 15 20:33:39 mds1 kernel: Call Trace:
Jan 15 20:33:39 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 20:33:39 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0eeb14c>] osp_precreate_reserve+0x5dc/0x1ef0 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0ee35c2>] ? osp_object_free+0x182/0x190 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0ee4b75>] osp_declare_object_create+0x155/0x4f0 [osp]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2f2dd>] lod_qos_declare_object_on+0xed/0x480 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e30ddf>] lod_alloc_rr.clone.2+0x66f/0xde0 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e325f4>] lod_qos_prep_create+0x10a4/0x1aa0 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa08fddfa>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0901b32>] ? fld_server_lookup+0x72/0x430 [fld]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2bb2b>] lod_declare_striped_object+0x14b/0x880 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e2c721>] lod_declare_object_create+0x4c1/0x790 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e878ef>] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffff811a58f7>] ? generic_getxattr+0x87/0x90
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e96eae>] mdd_declare_create+0x4e/0x870 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e953cf>] ? mdd_linkea_prepare+0x24f/0x4e0 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e97e91>] mdd_create+0x7c1/0x1730 [mdd]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0c99787>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0e28560>] ? lod_index_lookup+0x0/0x30 [lod]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d67dc8>] mdo_create+0x18/0x50 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d72031>] mdt_reint_open+0x1351/0x20a0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0432e16>] ? upcall_cache_get_entry+0x296/0x880 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0597600>] ? lu_ucred_global_init+0x0/0x30 [obdclass]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d5aeb1>] mdt_reint_rec+0x41/0xe0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d42c93>] mdt_reint_internal+0x4c3/0x780 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d4321d>] mdt_intent_reint+0x1ed/0x520 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 20:33:39 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 20:33:39 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 15 20:33:39 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 15 20:33:39 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 20:33:39 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 20:33:39 mds1 kernel: 
Jan 15 20:35:19 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:35:19 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:35:34 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 15 20:35:34 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 15 20:35:39 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 20:35:53 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation obd_ping failed with -107.
Jan 15 20:35:53 mds1 kernel: LustreError: Skipped 17 previous similar messages
Jan 15 20:35:53 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:35:53 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 15 20:36:10 mds1 kernel: LNet: Service thread pid 5487 was inactive for 218.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 15 20:36:10 mds1 kernel: Pid: 5487, comm: mdt00_004
Jan 15 20:36:10 mds1 kernel: 
Jan 15 20:36:10 mds1 kernel: Call Trace:
Jan 15 20:36:10 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 15 20:36:10 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06cc8a0>] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06d10b1>] ldlm_completion_ast+0x4b1/0x920 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06d0810>] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d40ccb>] mdt_object_lock0+0x33b/0xaf0 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d41544>] mdt_object_lock+0x14/0x20 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d4f359>] mdt_getattr_name_lock+0xe19/0x1990 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06f8f85>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa071ff06>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06fb214>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d50169>] mdt_intent_getattr+0x299/0x480 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 15 20:36:10 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 15 20:36:10 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 15 20:36:10 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 15 20:36:10 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 15 20:36:10 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 15 20:36:10 mds1 kernel: 
Jan 15 20:36:10 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389836170.5487
Jan 15 20:36:18 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 15 20:36:59 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:36:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:36:59 mds1 kernel: LNet: Service thread pid 4069 completed after 400.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jan 15 20:36:59 mds1 kernel: LNet: Service thread pid 5646 completed after 400.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jan 15 20:38:39 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:38:39 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:40:19 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:40:28 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 15 20:40:28 mds1 kernel: LustreError: Skipped 30 previous similar messages
Jan 15 20:41:59 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:42:32 mds1 kernel: Lustre: 3321:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389835796/real 1389835796]  req@ffff881019e0a400 x1457347449888776/t0(0) o6->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 664/432 e 0 to 1 dl 1389836552 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 20:42:32 mds1 kernel: Lustre: 3321:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 15 20:43:39 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:45:19 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:46:59 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:46:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 20:49:13 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_connect failed with -16.
Jan 15 20:49:13 mds1 kernel: LustreError: Skipped 62 previous similar messages
Jan 15 20:49:40 mds1 kernel: Lustre: emc-OST000d-osc-MDT0000: Connection to emc-OST000d (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:49:51 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection restored to emc-OST0016 (at 192.168.1.44@o2ib)
Jan 15 20:49:51 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.44@o2ib)
Jan 15 20:49:51 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 20:50:19 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: slow creates, last=[0x100070000:0x4181:0x0], next=[0x100070000:0x4181:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:50:19 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 15 20:51:59 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389837103/real 0]  req@ffff880815b13c00 x1457347449918804/t0(0) o8->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389837119 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:51:59 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 15 20:52:54 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389837153/real 0]  req@ffff880826020400 x1457347449919692/t0(0) o8->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389837174 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 20:53:51 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 06ad27f9-7bf7-6b94-ae3a-81f656485d22 (at 192.168.1.55@o2ib) in 245 seconds. I think it's dead, and I am evicting it. exp ffff880830110800, cur 1389837231 expire 1389837081 last 1389836986
Jan 15 20:53:51 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client d3698c71-399d-258a-bf5f-0fc22495db0f (at 192.168.1.56@o2ib) in 235 seconds. I think it's dead, and I am evicting it. exp ffff88101199a400, cur 1389837231 expire 1389837081 last 1389836996
Jan 15 20:55:20 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: slow creates, last=[0x100060000:0x4141:0x0], next=[0x100060000:0x4141:0x0], reserved=0, syn_changes=62, syn_rpc_in_progress=6, status=-19
Jan 15 20:55:20 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 15 20:55:28 mds1 kernel: Lustre: MGS: haven't heard from client 69c85377-b60e-ea0d-f22a-5e8e7a5c5c70 (at 192.168.1.46@o2ib) in 246 seconds. I think it's dead, and I am evicting it. exp ffff880ff7bb6000, cur 1389837328 expire 1389837178 last 1389837082
Jan 15 20:55:28 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 20:57:00 mds1 kernel: LustreError: 4069:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8810313a4000 ns: mdt-emc-MDT0000_UUID lock: ffff881011e8a940/0x34b17884aac35258 lrc: 3/0,0 mode: CR/CR res: [0x80000a041:0x23:0x0].0 bits 0x9 rrc: 1 type: IBT flags: 0x50200000000000 nid: 192.168.1.54@o2ib remote: 0xaebdb5669be1bcea expref: 3 pid: 4069 timeout: 0 lvb_type: 0
Jan 15 20:59:38 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_connect failed with -16.
Jan 15 20:59:38 mds1 kernel: LustreError: Skipped 62 previous similar messages
Jan 15 20:59:59 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:59:59 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 20:59:59 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection to emc-OST0016 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 20:59:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 15 21:00:04 mds1 kernel: Lustre: MGS: Client 831670c8-a1f9-ccff-4f82-c1d4d15d54a5 (at 192.168.1.46@o2ib) reconnecting
Jan 15 21:00:14 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.46@o2ib)
Jan 15 21:00:14 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 15 21:00:57 mds1 kernel: Lustre: 3318:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389837650/real 0]  req@ffff881009800c00 x1457347449928644/t0(0) o13->emc-OST0004-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389837657 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 21:00:57 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection to emc-OST0004 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 21:00:57 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 15 21:01:03 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389837657/real 0]  req@ffff880815a6b800 x1457347449928756/t0(0) o8->emc-OST0003-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389837663 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 21:01:03 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 15 21:01:58 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389837707/real 0]  req@ffff880831239c00 x1457347449929312/t0(0) o8->emc-OST0005-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389837718 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 21:01:58 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Jan 15 21:02:19 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:02:19 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0003_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:02:19 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000b_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:02:19 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 21:02:21 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0007_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:02:21 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 15 21:02:26 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection restored to emc-OST000b (at 192.168.1.46@o2ib)
Jan 15 21:02:26 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 21:03:11 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:03:11 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 21:04:01 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 15 21:04:01 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 15 21:04:20 mds1 kernel: Lustre: MGS: haven't heard from client 7c11bd17-4e5b-2f31-ad63-fd2a7e44743d (at 192.168.1.44@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff88100e7b3800, cur 1389837860 expire 1389837710 last 1389837631
Jan 15 21:06:01 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389837190/real 1389837190]  req@ffff8808310a2400 x1457347449920420/t0(0) o400->emc-OST000f-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 224/224 e 0 to 1 dl 1389837961 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 15 21:06:01 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 15 21:12:53 mds1 kernel: Lustre: 3312:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389838366/real 0]  req@ffff880815b9a000 x1457347449942160/t0(0) o13->emc-OST0001-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389838373 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 15 21:12:53 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 21:12:53 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 15 21:12:53 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection to emc-OST0005 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 21:12:56 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 21:12:56 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 21:13:27 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389838401/real 1389838401]  req@ffff880831185400 x1457347449942356/t0(0) o8->emc-OST0008-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389838407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 15 21:13:27 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 45 previous similar messages
Jan 15 21:13:57 mds1 kernel: Lustre: MGS: Client 7842f392-57ce-e079-de01-982c47ded436 (at 192.168.1.44@o2ib) reconnecting
Jan 15 21:13:57 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0014_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 15 21:13:57 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 15 21:14:07 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection restored to emc-OST000b (at 192.168.1.44@o2ib)
Jan 15 21:14:07 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 15 21:14:49 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000b_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 15 21:14:49 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0012_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 15 21:14:49 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0014_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 15 21:14:49 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 15 21:14:49 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 15 21:16:04 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 15 21:16:04 mds1 kernel: Lustre: Skipped 28 previous similar messages
Jan 15 21:16:29 mds1 kernel: Lustre: MGS: haven't heard from client 831670c8-a1f9-ccff-4f82-c1d4d15d54a5 (at 192.168.1.46@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff880815b00800, cur 1389838589 expire 1389838439 last 1389838341
Jan 15 21:16:29 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 15 21:16:41 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0011_UUID (at 192.168.1.46@o2ib) in 260 seconds. I think it's dead, and I am evicting it. exp ffff88103409b000, cur 1389838601 expire 1389838451 last 1389838341
Jan 15 21:18:37 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389837961/real 1389837961]  req@ffff8808310a2400 x1457347449934552/t0(0) o400->emc-OST000f-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 224/224 e 0 to 1 dl 1389838717 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 15 21:18:37 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 45 previous similar messages
Jan 15 21:20:47 mds1 kernel: LustreError: 11-0: emc-OST0014-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 15 21:20:47 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 21:20:47 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 21:20:47 mds1 kernel: LustreError: Skipped 18 previous similar messages
Jan 15 21:20:56 mds1 kernel: Lustre: MGS: Client d942e531-837e-7244-bc49-a2eefd1297c1 (at 192.168.1.46@o2ib) reconnecting
Jan 15 21:21:04 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection restored to emc-OST0010 (at 192.168.1.46@o2ib)
Jan 15 21:21:04 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 15 21:31:13 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389838717/real 1389838717]  req@ffff8808310a2400 x1457347449948124/t0(0) o400->emc-OST000f-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 224/224 e 0 to 1 dl 1389839473 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 15 21:31:13 mds1 kernel: LustreError: 167-0: emc-OST000f-osc-MDT0000: This client was evicted by emc-OST000f; in progress operations using this service will fail.
Jan 15 21:31:13 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection restored to emc-OST000f (at 192.168.1.46@o2ib)
Jan 15 21:31:13 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 22:17:22 mds1 kernel: LustreError: 11-0: emc-OST0005-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 15 22:17:22 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:17:22 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 15 22:17:22 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 15 22:17:40 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection restored to emc-OST0005 (at 192.168.1.46@o2ib)
Jan 15 22:32:32 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client d6316c01-1cd0-0ee8-a7f0-2f82eb7c4637 (at 192.168.1.55@o2ib) in 237 seconds. I think it's dead, and I am evicting it. exp ffff88081500e400, cur 1389843152 expire 1389843002 last 1389842915
Jan 15 22:32:32 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 15 22:34:11 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 1cc558cc-61db-fe15-25bf-8fc25df3c94f (at 192.168.1.54@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff8810119b9400, cur 1389843251 expire 1389843101 last 1389843003
Jan 15 22:40:00 mds1 kernel: LustreError: 11-0: emc-OST0004-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 15 22:40:00 mds1 kernel: LustreError: Skipped 9 previous similar messages
Jan 15 22:40:00 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection to emc-OST0004 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:40:00 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 22:40:00 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:40:00 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 15 22:40:05 mds1 kernel: Lustre: MGS: Client e464ab33-17b9-31db-2b39-587ab91e625c (at 192.168.1.44@o2ib) reconnecting
Jan 15 22:40:15 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection restored to emc-OST0007 (at 192.168.1.44@o2ib)
Jan 15 22:40:15 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 15 22:40:15 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection restored to emc-OST000a (at 192.168.1.44@o2ib)
Jan 15 22:40:15 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 22:40:28 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 15 22:40:28 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 15 22:40:28 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:40:28 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 15 22:40:29 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection to emc-OST0010 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:40:43 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection restored to emc-OST0013 (at 192.168.1.44@o2ib)
Jan 15 22:50:13 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 15 22:50:13 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:50:13 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection to emc-OST0013 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 15 22:50:13 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 15 22:50:13 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 15 22:50:13 mds1 kernel: LustreError: Skipped 16 previous similar messages
Jan 15 22:50:22 mds1 kernel: Lustre: MGS: Client c97ee32e-8d38-41c6-0ed4-2a42a1c6cc6a (at 192.168.1.46@o2ib) reconnecting
Jan 15 22:50:29 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 15 22:50:29 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 16 02:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:02 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:03 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:07 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:08 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:08 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:12 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:13 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:13 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:17 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:18 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:18 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:22 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:23 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:23 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:27 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:28 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:28 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:32 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:33 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:33 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:37 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:38 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:38 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:38 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:42 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:43 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:43 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:43 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:47 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:48 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:48 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:52 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:53 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:53 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:53 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:57 mds1 multipathd: 8:32: reinstated
Jan 16 02:24:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:24:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:24:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:24:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:24:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:24:58 mds1 multipathd: 8:80: reinstated
Jan 16 02:24:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:24:58 mds1 multipathd: 8:32: mark as failed
Jan 16 02:24:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:24:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:24:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:24:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:24:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:24:58 mds1 multipathd: 8:80: mark as failed
Jan 16 02:24:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:24:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:24:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:02 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:03 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:03 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:03 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:07 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:08 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:08 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:08 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:12 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:13 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:13 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:17 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:18 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:18 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:18 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:22 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:23 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:23 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:27 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:28 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:28 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:28 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:32 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:33 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:33 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:33 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:37 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:38 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:38 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:38 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:42 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:43 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:43 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:47 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:48 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:48 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:48 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:52 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:53 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:53 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:53 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:57 mds1 multipathd: 8:32: reinstated
Jan 16 02:25:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:25:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:25:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:25:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:25:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:25:58 mds1 multipathd: 8:80: reinstated
Jan 16 02:25:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:25:58 mds1 multipathd: 8:32: mark as failed
Jan 16 02:25:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:25:58 mds1 multipathd: 8:80: mark as failed
Jan 16 02:25:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:25:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:25:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:25:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:25:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:25:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:25:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:02 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:03 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:03 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:03 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:07 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:08 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:08 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:12 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:13 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:13 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:13 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:17 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:18 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:18 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:22 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:23 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:23 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:27 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:28 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:28 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:28 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:32 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:33 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:33 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:33 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:26:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:26:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:26:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:26:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:26:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:26:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:26:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:26:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:26:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:26:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:26:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:26:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:26:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:26:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:27:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:27:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:27:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:27:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:27:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:27:24 mds1 kernel: Call Trace:
Jan 16 02:27:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:27:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:27:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:27:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:27:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:27:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:27:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:27:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:27:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:27:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:27:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:27:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:27:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:27:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:27:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:27:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:27:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:27:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:27:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:27:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:27:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:27:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:27:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:27:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:27:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:27:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:27:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:27:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:27:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:27:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:27:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:27:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:27:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:27:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:27:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:28:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:28:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:28:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:28:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:28:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:28:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:28:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:28:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:28:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:28:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:28:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:28:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:28:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:29:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:29:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:29:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:29:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:29:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:29:24 mds1 kernel: Call Trace:
Jan 16 02:29:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:29:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:29:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:29:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:29:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:29:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:29:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:29:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:29:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:29:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:29:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:29:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:29:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:29:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:29:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:29:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:29:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:29:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:29:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:29:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:29:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:29:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:29:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:29:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:29:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:29:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:29:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:29:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:29:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:29:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:29:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:29:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:29:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:29:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:29:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:30:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:30:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:30:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:30:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:30:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:30:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:30:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:30:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:30:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:30:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:30:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:30:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:30:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:30:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:31:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:31:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:31:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:31:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:31:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:31:24 mds1 kernel: Call Trace:
Jan 16 02:31:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:31:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:31:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:31:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:31:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:31:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:31:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:31:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:31:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:31:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:31:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:31:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:31:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:31:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:31:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:31:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:31:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:31:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:31:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:31:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:31:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:31:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:31:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:31:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:31:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:31:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:31:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:31:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:31:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:31:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:31:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:31:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:31:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:31:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:31:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:32:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:32:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:32:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:32:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:32:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:32:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:32:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:32:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:32:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:32:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:32:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:32:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:32:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:32:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:33:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:33:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:33:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:33:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:33:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:33:24 mds1 kernel: Call Trace:
Jan 16 02:33:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:33:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:33:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:33:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:33:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:33:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:33:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:33:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:33:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:33:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:33:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:33:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:33:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:33:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:33:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:33:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:33:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:33:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:33:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:33:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:33:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:33:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:33:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:33:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:33:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:33:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:33:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:33:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:33:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:33:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:33:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:33:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:33:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:33:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:33:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:34:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:34:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:34:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:34:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:34:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:34:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:34:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:34:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:34:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:34:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:34:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:34:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:34:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:34:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:35:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:35:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:35:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:35:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:35:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:35:24 mds1 kernel: Call Trace:
Jan 16 02:35:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:35:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:35:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:35:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:35:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:35:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:35:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:35:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:35:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:35:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:35:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:35:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:35:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:35:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:35:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:35:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:35:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:35:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:35:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:35:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:35:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:35:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:35:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:35:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:35:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:35:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:35:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:35:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:35:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:35:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:35:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:35:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:35:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:35:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:35:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:35:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:35:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:35:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:36:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:36:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:36:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:36:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:36:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:36:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:36:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:36:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:36:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:36:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:36:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:36:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:36:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:36:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:36:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:36:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:36:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:37:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:37:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:37:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:37:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:37:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:37:24 mds1 kernel: Call Trace:
Jan 16 02:37:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:37:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:37:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:37:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:37:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:37:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:37:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:37:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:37:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:37:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:37:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:37:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:37:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:37:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:37:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:37:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:37:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:37:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:37:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:37:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:37:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:37:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:37:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:37:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:37:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:37:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:37:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:37:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:37:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:37:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:37:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:37:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:37:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:37:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:37:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:37:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:37:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:37:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:38:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:38:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:38:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:38:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:38:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:38:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:38:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:38:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:38:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:38:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:38:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:38:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:38:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:38:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:38:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:38:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:38:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:39:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:39:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:39:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:39:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:39:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:39:24 mds1 kernel: Call Trace:
Jan 16 02:39:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:39:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:39:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:39:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:39:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:39:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:39:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:39:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:39:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:39:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:39:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:39:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:39:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:39:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:39:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:39:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:39:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:39:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:39:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:39:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:39:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:39:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:39:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:39:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:39:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:39:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:39:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:39:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:39:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:39:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:39:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:39:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:39:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:39:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:39:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:39:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:39:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:40:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:40:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:40:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:40:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:40:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:40:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:40:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:40:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:40:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:40:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:40:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:40:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:40:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:40:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:40:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:40:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:40:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:41:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:41:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:41:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:41:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:41:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:41:24 mds1 kernel: Call Trace:
Jan 16 02:41:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:41:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:41:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:41:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:41:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:41:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:41:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:41:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:41:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:41:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:41:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:41:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:41:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:41:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:41:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:41:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:41:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:41:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:41:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:41:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:41:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:41:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:41:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:41:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:41:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:41:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:41:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:41:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:41:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:41:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:41:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:41:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:41:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:41:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:41:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:41:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:41:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:41:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:42:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:42:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:42:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:42:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:42:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:42:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:42:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:42:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:42:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:42:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:42:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:42:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:42:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:42:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:42:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:42:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:42:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:43:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:43:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:43:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:43:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:43:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:43:24 mds1 kernel: Call Trace:
Jan 16 02:43:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:43:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:43:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:43:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:43:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:43:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:43:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:43:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:43:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:43:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:43:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:43:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:43:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:43:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:43:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:43:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:43:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:43:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:43:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:43:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:43:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:43:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:43:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:43:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:43:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:43:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:43:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:43:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:43:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:43:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:43:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:43:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:43:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:43:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:43:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:43:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:43:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:43:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:44:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:44:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:44:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:44:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:44:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:44:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:44:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:44:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:44:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:44:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:44:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:44:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:44:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:44:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:44:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:44:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:44:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:24 mds1 kernel: INFO: task jbd2/dm-0-8:4336 blocked for more than 120 seconds.
Jan 16 02:45:24 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 02:45:24 mds1 kernel: jbd2/dm-0-8   D 0000000000000006     0  4336      2 0x00000080
Jan 16 02:45:24 mds1 kernel: ffff88082606dc20 0000000000000046 ffff88082606dbe0 ffffffffa00043fc
Jan 16 02:45:24 mds1 kernel: ffff88082d2d4178 ffff880832b82e00 ffff88082606dcc0 0000000000000002
Jan 16 02:45:24 mds1 kernel: ffff88082e83dab8 ffff88082606dfd8 000000000000fb88 ffff88082e83dab8
Jan 16 02:45:24 mds1 kernel: Call Trace:
Jan 16 02:45:24 mds1 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jan 16 02:45:24 mds1 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Jan 16 02:45:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:45:24 mds1 kernel: [<ffffffff8150ed03>] io_schedule+0x73/0xc0
Jan 16 02:45:24 mds1 kernel: [<ffffffff811b63a0>] sync_buffer+0x40/0x50
Jan 16 02:45:24 mds1 kernel: [<ffffffff8150f6bf>] __wait_on_bit+0x5f/0x90
Jan 16 02:45:24 mds1 kernel: [<ffffffff811b6360>] ? sync_buffer+0x0/0x50
Jan 16 02:45:24 mds1 kernel: [<ffffffff8150f768>] out_of_line_wait_on_bit+0x78/0x90
Jan 16 02:45:24 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 02:45:24 mds1 kernel: [<ffffffff811b6356>] __wait_on_buffer+0x26/0x30
Jan 16 02:45:24 mds1 kernel: [<ffffffffa010b206>] jbd2_journal_commit_transaction+0xa86/0x15a0 [jbd2]
Jan 16 02:45:24 mds1 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
Jan 16 02:45:24 mds1 kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Jan 16 02:45:24 mds1 kernel: [<ffffffffa0111308>] kjournald2+0xb8/0x220 [jbd2]
Jan 16 02:45:24 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 02:45:24 mds1 kernel: [<ffffffffa0111250>] ? kjournald2+0x0/0x220 [jbd2]
Jan 16 02:45:24 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 02:45:24 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 02:45:24 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 02:45:24 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 02:45:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:45:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:45:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:45:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:45:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:45:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:45:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:45:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:45:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:45:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:45:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:45:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:45:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:45:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:45:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:45:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:45:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:45:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:46:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:46:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:46:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:46:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:46:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:46:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:46:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:46:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:46:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:46:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:46:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:46:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:46:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:46:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:46:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:46:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:46:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:47:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:47:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:47:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:47:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:47:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:47:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:47:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:47:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:47:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:47:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:47:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:47:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:47:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:47:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:47:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:47:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:47:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:48:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:48:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:48:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:48:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:48:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:48:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:48:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:48:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:48:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:48:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:48:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:48:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:48:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:48:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:48:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:48:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:48:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:49:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:49:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:49:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:49:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:49:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:49:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:49:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:49:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:49:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:49:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:49:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:49:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:49:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:49:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:49:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:49:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:49:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:50:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:50:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:50:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:50:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:50:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:50:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:50:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:50:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:50:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:50:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:50:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:50:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:50:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:50:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:50:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:50:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:50:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:51:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:51:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:51:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:51:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:51:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:51:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:51:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:51:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:51:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:51:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:51:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:51:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:51:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:51:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:51:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:51:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:51:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:52:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:52:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:52:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:52:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:52:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:52:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:52:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:52:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:52:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:52:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:52:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:52:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:52:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:52:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:52:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:52:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:52:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:53:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:53:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:53:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:53:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:53:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:53:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:53:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:53:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:53:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:53:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:53:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:53:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:53:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:53:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:53:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:53:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:53:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:54:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:54:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:54:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:54:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:54:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:54:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:54:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:54:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:54:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:54:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:54:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:54:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:54:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:54:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:54:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:54:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:54:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:55:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:55:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:55:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:55:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:55:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:55:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:55:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:55:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:55:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:55:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:55:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:55:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:55:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:55:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:55:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:55:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:55:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:19 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:56:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:56:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:56:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:56:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:56:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:56:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:56:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:56:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:56:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:56:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:56:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:56:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:56:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:56:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:56:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:56:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:56:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:10 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:24 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:57:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:57:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:57:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:57:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:57:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:57:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:57:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:57:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:57:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:57:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:57:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:57:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:57:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:57:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:57:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:57:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:57:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:00 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:04 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:15 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:30 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:34 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:40 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:45 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:49 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:54 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:58:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:58:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:58:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:58:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:58:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:58:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:58:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:58:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:58:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:58:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:58:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:58:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:58:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:58:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:58:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:58:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:58:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:03 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:04 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:04 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:05 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:08 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:09 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:09 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:09 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:13 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:14 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:14 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:14 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:18 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:19 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:19 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:20 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:23 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:24 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:24 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:25 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:28 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:29 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:29 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:29 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:33 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:34 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:34 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:35 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:38 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:39 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:39 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:39 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:43 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:44 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:44 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:44 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:48 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:49 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:49 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:50 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:53 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:54 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:54 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:55 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:58 mds1 multipathd: 8:32: reinstated
Jan 16 02:59:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 02:59:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 02:59:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 02:59:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 02:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 02:59:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 02:59:59 mds1 multipathd: 8:80: reinstated
Jan 16 02:59:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 02:59:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:59 mds1 multipathd: 8:32: mark as failed
Jan 16 02:59:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 02:59:59 mds1 multipathd: 8:80: mark as failed
Jan 16 02:59:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 02:59:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 02:59:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 02:59:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 02:59:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 02:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 02:59:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 02:59:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:03 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:04 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:04 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:08 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:09 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:09 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:13 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:14 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:14 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:18 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:19 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:19 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:19 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:23 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:24 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:24 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:28 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:29 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:29 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:29 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:33 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:34 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:34 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:34 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:38 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:39 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:39 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:39 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:43 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:44 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:44 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:48 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:49 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:49 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:49 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:53 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:00:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:54 mds1 multipathd: 8:80: reinstated
Jan 16 03:00:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:00:54 mds1 multipathd: 8:32: mark as failed
Jan 16 03:00:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:00:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:00:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:00:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:00:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:00:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:00:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:00:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:00:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:00:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:00:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:00:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:01:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:01:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:01:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:01:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:01:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:01:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:01:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:01:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:01:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:01:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:01:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:01:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:01:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:01:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:02:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:02:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:02:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:02:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:02:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:02:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:02:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:02:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:02:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:02:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:02:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:02:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:02:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:03:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:03:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:03:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:03:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:03:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:03:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:03:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:03:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:03:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:03:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:03:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:03:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:03:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:04:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:04:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:04:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:04:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:04:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:04:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:04:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:04:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:04:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:04:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:04:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:04:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:04:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:04:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:05:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:05:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:05:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:05:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:05:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:05:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:05:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:05:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:05:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:05:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:05:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:05:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:05:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:05:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:06:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:06:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:06:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:06:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:06:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:06:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:06:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:06:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:06:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:06:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:06:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:06:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:06:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:06:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:07:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:07:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:07:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:07:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:07:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:07:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:07:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:07:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:07:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:07:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:07:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:07:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:07:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:07:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:07:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:07:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:07:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:08:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:08:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:08:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:08:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:08:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:08:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:08:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:08:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:08:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:08:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:08:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:08:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:08:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:08:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:08:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:08:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:08:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:09:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:09:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:09:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:09:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:09:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:09:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:09:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:09:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:09:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:09:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:09:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:09:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:09:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:09:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:09:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:09:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:09:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:09:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:10:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:10:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:10:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:10:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:10:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:10:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:10:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:10:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:10:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:10:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:10:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:10:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:10:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:10:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:10:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:10:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:10:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:10:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:11:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:11:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:11:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:11:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:11:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:11:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:11:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:11:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:11:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:11:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:11:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:11:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:11:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:11:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:11:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:11:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:11:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:11:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:12:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:12:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:12:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:12:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:12:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:12:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:12:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:12:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:12:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:12:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:12:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:12:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:12:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:12:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:12:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:12:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:12:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:12:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:13:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:13:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:13:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:13:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:13:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:13:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:13:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:13:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:13:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:13:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:13:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:13:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:13:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:13:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:13:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:13:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:13:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:13:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:14:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:14:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:14:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:14:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:14:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:14:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:14:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:14:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:14:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:14:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:14:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:14:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:14:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:14:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:14:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:14:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:14:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:14:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:15:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:15:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:15:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:15:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:15:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:15:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:15:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:15:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:15:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:15:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:15:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:15:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:15:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:15:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:15:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:15:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:15:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:15:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:16:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:16:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:16:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:16:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:16:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:16:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:16:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:16:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:16:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:16:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:16:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:16:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:16:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:16:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:16:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:16:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:16:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:16:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:17:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:17:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:17:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:17:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:17:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:17:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:17:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:17:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:17:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:17:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:17:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:17:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:17:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:17:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:17:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:17:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:17:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:17:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:18:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:18:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:18:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:18:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:18:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:18:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:18:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:18:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:18:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:18:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:18:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:18:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:18:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:18:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:18:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:18:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:18:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:18:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:19:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:19:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:19:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:19:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:19:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:19:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:19:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:19:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:19:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:19:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:19:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:19:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:19:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:19:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:19:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:19:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:19:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:19:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:20:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:20:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:20:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:20:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:20:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:20:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:20:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:20:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:20:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:20:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:20:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:20:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:20:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:20:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:20:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:20:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:20:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:20:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:21:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:21:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:21:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:21:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:21:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:21:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:21:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:21:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:21:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:21:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:21:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:21:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:21:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:21:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:21:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:21:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:21:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:21:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:50 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:22:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:22:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:22:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:22:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:22:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:22:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:22:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:22:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:22:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:22:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:22:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:22:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:22:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:22:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:22:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:22:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:22:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:22:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:25 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:23:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:23:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:23:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:23:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:23:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:23:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:23:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:23:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:23:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:23:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:23:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:23:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:23:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:23:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:23:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:23:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:23:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:23:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:24:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:24:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:24:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:24:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:24:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:24:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:24:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:24:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:24:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:24:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:24:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:24:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:24:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:24:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:24:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:24:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:24:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:24:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:25:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:25:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:25:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:25:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:25:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:25:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:25:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:25:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:25:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:25:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:25:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:25:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:25:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:25:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:25:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:25:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:25:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:25:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:26:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:26:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:26:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:26:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:26:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:26:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:26:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:26:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:26:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:26:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:26:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:26:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:26:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:26:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:26:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:26:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:26:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:26:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:27:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:27:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:27:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:27:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:27:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:27:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:27:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:27:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:27:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:27:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:27:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:27:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:27:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:27:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:27:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:27:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:27:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:27:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:35 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:28:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:28:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:28:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:28:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:28:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:28:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:28:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:28:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:28:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:28:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:28:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:28:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:28:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:28:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:28:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:28:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:28:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:28:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:29:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:29:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:29:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:29:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:29:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:29:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:29:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:29:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:29:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:29:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:29:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:29:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:29:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:29:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:29:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:29:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:29:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:29:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:30:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:30:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:30:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:30:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:30:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:30:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:30:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:30:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:30:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:30:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:30:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:30:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:30:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:30:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:30:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:30:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:30:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:30:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:31:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:31:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:31:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:31:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:31:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:31:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:31:55 mds1 multipathd: 8:80: mark as failed
Jan 16 03:31:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:31:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:31:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:31:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:31:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:31:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:31:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:31:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:31:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:31:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:31:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:15 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:20 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:32:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:32:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:32:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:32:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:32:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:32:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:32:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:32:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:32:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:32:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:32:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:32:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:32:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:32:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:32:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:32:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:32:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:32:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:00 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:05 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:33:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:33:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:33:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:33:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:33:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:33:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:33:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:33:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:33:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:33:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:33:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:33:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:33:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:33:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:33:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:33:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:33:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:33:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:24 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:25 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:25 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:29 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:30 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:30 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:30 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:34 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:35 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:35 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:39 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:40 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:40 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:40 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:44 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:45 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:45 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:45 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:49 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:50 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:50 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:54 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:34:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:55 mds1 multipathd: 8:80: reinstated
Jan 16 03:34:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:34:55 mds1 multipathd: 8:32: mark as failed
Jan 16 03:34:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:34:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:34:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:34:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:34:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:34:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:34:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:34:59 mds1 multipathd: 8:32: reinstated
Jan 16 03:34:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:34:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:34:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:34:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:34:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:34:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:34:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:00 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:00 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:04 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:05 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:05 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:09 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:10 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:10 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:10 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:14 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:15 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:15 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:19 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:20 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:20 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:35:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:35:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:35:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:35:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:35:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:35:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:35:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:35:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:35:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:35:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:35:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:35:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:35:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:35:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:35:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:35:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:36:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:36:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:36:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:36:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:36:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:36:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:36:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:36:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:36:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:36:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:36:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:36:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:36:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:36:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:36:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:36:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:36:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:37:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:37:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:37:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:37:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:37:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:37:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:37:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:37:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:37:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:37:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:37:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:37:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:37:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:37:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:37:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:37:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:37:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:38:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:38:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:38:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:38:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:38:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:38:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:38:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:38:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:38:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:38:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:38:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:38:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:38:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:38:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:38:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:38:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:38:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:39:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:39:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:39:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:39:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:39:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:39:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:39:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:39:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:39:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:39:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:39:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:39:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:39:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:39:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:39:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:39:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:39:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:40:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:40:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:40:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:40:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:40:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:40:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:40:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:40:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:40:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:40:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:40:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:40:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:40:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:40:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:40:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:40:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:40:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:41:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:41:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:41:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:41:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:41:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:41:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:41:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:41:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:41:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:41:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:41:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:41:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:41:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:41:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:41:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:41:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:41:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:42:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:42:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:42:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:42:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:42:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:42:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:42:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:42:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:42:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:42:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:42:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:42:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:42:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:42:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:42:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:42:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:42:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:42:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:43:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:43:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:43:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:43:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:43:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:43:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:43:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:43:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:43:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:43:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:43:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:43:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:43:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:43:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:43:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:43:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:43:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:43:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:44:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:44:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:44:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:44:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:44:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:44:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:44:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:44:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:44:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:44:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:44:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:44:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:44:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:44:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:44:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:44:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:44:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:44:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:45:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:45:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:45:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:45:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:45:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:45:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:45:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:45:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:45:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:45:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:45:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:45:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:45:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:45:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:45:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:45:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:45:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:45:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:46:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:46:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:46:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:46:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:46:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:46:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:46:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:46:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:46:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:46:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:46:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:46:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:46:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:46:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:46:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:46:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:46:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:46:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:47:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:47:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:47:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:47:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:47:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:47:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:47:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:47:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:47:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:47:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:47:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:47:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:47:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:47:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:47:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:47:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:47:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:47:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:48:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:48:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:48:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:48:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:48:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:48:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:48:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:48:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:48:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:48:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:48:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:48:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:48:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:48:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:48:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:48:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:48:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:48:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:49:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:49:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:49:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:49:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:49:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:49:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:49:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:49:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:49:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:49:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:49:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:49:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:49:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:49:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:49:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:49:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:49:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:49:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:50:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:50:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:50:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:50:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:50:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:50:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:50:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:50:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:50:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:50:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:50:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:50:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:50:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:50:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:50:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:50:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:50:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:50:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:51:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:51:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:51:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:51:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:51:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:51:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:51:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:51:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:51:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:51:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:51:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:51:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:51:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:51:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:51:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:51:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:51:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:51:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:52:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:52:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:52:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:52:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:52:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:52:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:52:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:52:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:52:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:52:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:52:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:52:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:52:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:52:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:52:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:52:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:52:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:52:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:53:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:53:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:53:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:53:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:53:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:53:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:53:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:53:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:53:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:53:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:53:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:53:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:53:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:53:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:53:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:53:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:53:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:53:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:21 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:54:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:54:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:54:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:54:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:54:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:54:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:54:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:54:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:54:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:54:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:54:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:54:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:54:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:54:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:54:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:54:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:54:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:54:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:55:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:55:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:55:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:55:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:55:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:55:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:55:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:55:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:55:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:55:56 mds1 multipathd: 8:80: mark as failed
Jan 16 03:55:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:55:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:55:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:55:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:55:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:55:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:55:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:55:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:56:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:56:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:56:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:56:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:56:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:56:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:56:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:56:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:56:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:56:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:56:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:56:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:56:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:56:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:56:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:56:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:56:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:56:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:01 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:06 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:17 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:31 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:41 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:57:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:57:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:57:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:57:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:57:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:57:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:57:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:57:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:57:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:57:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:57:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:57:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:57:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:57:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:57:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:57:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:57:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:57:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:12 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:26 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:36 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:46 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:52 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:58:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:58:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:58:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:58:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:58:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:58:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:58:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:58:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:58:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:58:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:58:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:58:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:58:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:58:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:58:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:58:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:58:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:58:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:00 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:01 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:01 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:02 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:05 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:06 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:06 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:07 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:10 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:11 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:11 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:11 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:15 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:16 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:16 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:16 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:20 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:21 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:21 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:22 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:25 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:26 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:26 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:27 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:30 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:31 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:31 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:32 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:35 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:36 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:36 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:37 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:40 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:41 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:41 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:42 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:45 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:46 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:46 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:47 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:50 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:51 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:51 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:51 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 03:59:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:55 mds1 multipathd: 8:32: reinstated
Jan 16 03:59:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 03:59:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 03:59:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 03:59:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 03:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 03:59:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 03:59:56 mds1 multipathd: 8:80: reinstated
Jan 16 03:59:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 03:59:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:56 mds1 multipathd: 8:32: mark as failed
Jan 16 03:59:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 03:59:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 03:59:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 03:59:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 03:59:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 03:59:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 03:59:57 mds1 multipathd: 8:80: mark as failed
Jan 16 03:59:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 03:59:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:21 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:26 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:36 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:51 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:00:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:00:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:00:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:00:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:00:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:00:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:00:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:00:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:00:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:00:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:00:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:00:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:00:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:00:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:00:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:00:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:00:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:00:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:01:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:01:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:01:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:01:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:01:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:01:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:01:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:01:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:01:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:01:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:01:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:01:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:01:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:01:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:01:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:01:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:01:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:01:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:06 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:31 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:51 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:02:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:02:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:02:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:02:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:02:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:02:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:02:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:02:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:02:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:02:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:02:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:02:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:02:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:02:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:02:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:02:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:02:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:02:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:06 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:21 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:26 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:36 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:46 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:51 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:03:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:03:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:03:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:03:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:03:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:03:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:03:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:03:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:03:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:03:56 mds1 multipathd: 8:80: mark as failed
Jan 16 04:03:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:03:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:03:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:03:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:03:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:03:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:03:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:03:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:04:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:04:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:04:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:04:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:04:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:04:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:04:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:04:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:04:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:04:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:04:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:04:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:04:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:04:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:04:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:04:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:04:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:04:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:06 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:31 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:36 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:46 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:05:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:05:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:05:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:05:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:05:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:05:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:05:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:05:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:05:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:05:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:05:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:05:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:05:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:05:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:05:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:05:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:05:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:05:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:21 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:06:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:06:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:06:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:06:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:06:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:06:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:06:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:06:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:06:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:06:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:06:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:06:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:06:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:06:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:06:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:06:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:06:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:06:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:26 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:31 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:07:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:07:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:07:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:07:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:07:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:07:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:07:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:07:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:07:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:07:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:07:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:07:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:07:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:07:56 mds1 multipathd: 8:80: mark as failed
Jan 16 04:07:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:07:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:07:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:06 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:20 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:21 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:21 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:21 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:25 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:26 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:26 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:30 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:31 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:31 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:35 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:36 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:36 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:40 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:41 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:41 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:41 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:45 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:46 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:46 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:50 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:51 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:51 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:08:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:55 mds1 multipathd: 8:32: reinstated
Jan 16 04:08:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:08:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:08:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:08:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:08:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:08:56 mds1 multipathd: 8:80: reinstated
Jan 16 04:08:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:08:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:56 mds1 multipathd: 8:32: mark as failed
Jan 16 04:08:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:08:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:08:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:08:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:08:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:08:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:08:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:08:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:08:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:00 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:01 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:01 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:01 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:05 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:06 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:06 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:10 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:11 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:11 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:11 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:15 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:16 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:16 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:16 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:09:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:09:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:09:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:09:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:09:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:09:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:09:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:09:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:09:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:09:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:09:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:09:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:09:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:09:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:09:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:09:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:09:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:09:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:10:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:10:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:10:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:10:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:10:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:10:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:10:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:10:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:10:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:10:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:10:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:10:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:10:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:10:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:10:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:10:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:10:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:10:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:11:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:11:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:11:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:11:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:11:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:11:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:11:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:11:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:11:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:11:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:11:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:11:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:11:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:11:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:11:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:11:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:11:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:11:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:12:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:12:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:12:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:12:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:12:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:12:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:12:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:12:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:12:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:12:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:12:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:12:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:12:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:12:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:12:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:12:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:12:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:12:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:13:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:13:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:13:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:13:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:13:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:13:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:13:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:13:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:13:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:13:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:13:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:13:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:13:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:13:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:13:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:13:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:13:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:13:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:14:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:14:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:14:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:14:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:14:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:14:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:14:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:14:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:14:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:14:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:14:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:14:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:14:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:14:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:14:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:14:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:14:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:14:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:15:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:15:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:15:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:15:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:15:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:15:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:15:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:15:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:15:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:15:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:15:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:15:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:15:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:15:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:15:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:15:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:15:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:15:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:15:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:16:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:16:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:16:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:16:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:16:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:16:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:16:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:16:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:16:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:16:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:16:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:16:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:16:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:16:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:16:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:16:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:16:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:16:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:16:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:17:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:17:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:17:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:17:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:17:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:17:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:17:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:17:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:17:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:17:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:17:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:17:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:17:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:17:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:17:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:17:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:17:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:17:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:17:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:18:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:18:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:18:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:18:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:18:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:18:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:18:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:18:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:18:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:18:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:18:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:18:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:18:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:18:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:18:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:18:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:18:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:18:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:19:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:19:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:19:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:19:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:19:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:19:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:19:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:19:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:19:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:19:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:19:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:19:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:19:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:19:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:19:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:19:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:19:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:19:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:20:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:20:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:20:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:20:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:20:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:20:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:20:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:20:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:20:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:20:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:20:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:20:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:20:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:20:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:20:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:20:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:20:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:20:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:21:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:21:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:21:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:21:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:21:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:21:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:21:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:21:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:21:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:21:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:21:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:21:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:21:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:21:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:21:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:21:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:21:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:21:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:22:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:22:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:22:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:22:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:22:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:22:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:22:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:22:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:22:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:22:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:22:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:22:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:22:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:22:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:22:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:22:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:22:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:22:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:23:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:23:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:23:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:23:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:23:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:23:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:23:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:23:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:23:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:23:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:23:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:23:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:23:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:23:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:23:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:23:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:23:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:23:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:24:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:24:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:24:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:24:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:24:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:24:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:24:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:24:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:24:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:24:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:24:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:24:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:24:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:24:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:24:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:24:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:24:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:25:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:25:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:25:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:25:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:25:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:25:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:25:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:25:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:25:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:25:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:25:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:25:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:25:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:25:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:25:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:25:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:25:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:26:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:26:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:26:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:26:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:26:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:26:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:26:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:26:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:26:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:26:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:26:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:26:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:26:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:26:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:26:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:26:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:26:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:26:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:26:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:27:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:27:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:27:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:27:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:27:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:27:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:27:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:27:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:27:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:27:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:27:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:27:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:27:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:27:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:27:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:27:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:27:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:27:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:27:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:28:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:28:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:28:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:28:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:28:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:28:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:28:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:28:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:28:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:28:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:28:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:28:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:28:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:28:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:28:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:28:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:28:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:28:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:28:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:29:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:29:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:29:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:29:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:29:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:29:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:29:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:29:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:29:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:29:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:29:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:29:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:29:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:29:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:29:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:29:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:29:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:29:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:29:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:30:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:30:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:30:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:30:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:30:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:30:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:30:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:30:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:30:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:30:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:30:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:30:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:30:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:30:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:30:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:30:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:30:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:30:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:30:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:31:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:31:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:31:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:31:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:31:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:31:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:31:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:31:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:31:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:31:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:31:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:31:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:31:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:31:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:31:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:31:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:31:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:31:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:31:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:47 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:32:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:32:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:32:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:32:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:32:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:32:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:32:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:32:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:32:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:32:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:32:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:32:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:32:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:32:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:32:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:32:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:32:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:32:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:32:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:33:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:33:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:33:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:33:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:33:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:33:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:33:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:33:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:33:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:33:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:33:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:33:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:33:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:33:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:33:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:33:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:33:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:33:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:33:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:34:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:34:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:34:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:34:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:34:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:34:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:34:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:34:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:34:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:34:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:34:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:34:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:34:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:34:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:34:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:34:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:34:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:34:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:34:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:35:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:35:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:35:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:35:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:35:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:35:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:35:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:35:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:35:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:35:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:35:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:35:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:35:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:35:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:35:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:35:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:35:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:35:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:36:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:36:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:36:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:36:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:36:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:36:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:36:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:36:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:36:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:36:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:36:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:36:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:36:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:36:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:36:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:36:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:36:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:36:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:17 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:37:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:37:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:37:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:37:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:37:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:37:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:37:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:37:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:37:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:37:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:37:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:37:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:37:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:37:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:37:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:37:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:37:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:37:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:07 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:52 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:38:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:38:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:38:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:38:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:38:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:38:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:38:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:38:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:38:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:38:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:38:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:38:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:38:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:38:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:38:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:38:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:38:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:38:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:02 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:39:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:39:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:39:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:39:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:39:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:39:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:39:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:39:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:39:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:39:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:39:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:39:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:39:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:39:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:39:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:39:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:39:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:39:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:12 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:32 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:40:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:40:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:40:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:40:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:40:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:40:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:40:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:40:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:40:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:40:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:40:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:40:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:40:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:40:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:40:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:40:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:40:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:40:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:01 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:02 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:02 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:06 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:07 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:07 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:11 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:12 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:12 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:16 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:17 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:17 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:21 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:22 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:22 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:22 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:26 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:27 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:27 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:27 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:31 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:32 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:32 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:36 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:37 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:37 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:37 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:41 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:42 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:42 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:42 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:46 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:47 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:47 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:51 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:52 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:52 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:56 mds1 multipathd: 8:32: reinstated
Jan 16 04:41:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:41:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:41:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:41:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:41:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:41:57 mds1 multipathd: 8:80: reinstated
Jan 16 04:41:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:41:57 mds1 multipathd: 8:32: mark as failed
Jan 16 04:41:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:41:57 mds1 multipathd: 8:80: mark as failed
Jan 16 04:41:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:41:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:41:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:41:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:41:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:41:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:41:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:41:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:42:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:42:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:42:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:42:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:42:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:42:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:42:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:42:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:42:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:42:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:42:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:42:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:42:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:42:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:42:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:42:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:42:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:43:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:43:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:43:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:43:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:43:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:43:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:43:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:43:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:43:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:43:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:43:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:43:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:43:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:43:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:43:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:43:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:43:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:43:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:44 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:44:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:44:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:44:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:44:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:44:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:44:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:44:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:44:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:44:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:44:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:44:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:44:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:44:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:44:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:44:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:44:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:44:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:45:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:45:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:45:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:45:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:45:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:45:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:45:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:45:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:45:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:45:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:45:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:45:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:45:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:45:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:45:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:45:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:45:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:45:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:46:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:46:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:46:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:46:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:46:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:46:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:46:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:46:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:46:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:46:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:46:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:46:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:46:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:46:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:46:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:46:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:46:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:46:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:47:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:47:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:47:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:47:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:47:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:47:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:47:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:47:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:47:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:47:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:47:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:47:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:47:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:47:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:47:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:47:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:47:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:47:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:44 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:48:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:48:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:48:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:48:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:48:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:48:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:48:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:48:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:48:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:48:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:48:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:48:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:48:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:48:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:48:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:48:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:48:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:48:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:44 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:49:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:49:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:49:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:49:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:49:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:49:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:49:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:49:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:49:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:49:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:49:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:49:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:49:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:49:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:49:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:49:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:49:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:49:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:50:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:50:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:50:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:50:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:50:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:50:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:50:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:50:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:50:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:50:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:50:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:50:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:50:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:50:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:50:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:50:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:50:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:50:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:51:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:51:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:51:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:51:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:51:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:51:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:51:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:51:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:51:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:51:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:51:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:51:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:51:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:51:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:51:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:51:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:51:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:51:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:48 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:52:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:52:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:52:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:52:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:52:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:52:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:52:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:52:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:52:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:52:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:52:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:52:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:52:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:52:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:52:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:52:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:52:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:53:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:53:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:53:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:53:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:53:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:53:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:53:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:53:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:53:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:53:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:53:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:53:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:53:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:53:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:53:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:53:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:53:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:28 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:54:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:54:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:54:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:54:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:54:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:54:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:54:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:54:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:54:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:54:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:54:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:54:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:54:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:54:58 mds1 multipathd: 8:80: mark as failed
Jan 16 04:54:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:54:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:54:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:33 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:53 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:55:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:55:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:55:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:55:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:55:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:55:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:55:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:55:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:55:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:55:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:55:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:55:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:55:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:55:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:55:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:55:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:55:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:08 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:18 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:56:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:56:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:56:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:56:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:56:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:56:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:56:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:56:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:56:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:56:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:56:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:56:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:56:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:56:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:56:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:56:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:56:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:24 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:39 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:57:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:57:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:57:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:57:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:57:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:57:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:57:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:57:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:57:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:57:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:57:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:57:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:57:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:57:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:57:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:57:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:57:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:03 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:13 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:44 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:58:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:58:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:58:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:58:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:58:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:58:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:58:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:58:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:58:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:58:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:58:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:58:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:58:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:58:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:58:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:58:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:58:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:02 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:03 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:03 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:04 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:07 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:08 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:08 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:09 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:12 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:13 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:13 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:14 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:17 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:18 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:18 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:19 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:22 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:23 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:23 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:23 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:27 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:28 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:28 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:29 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:32 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:33 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:33 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:34 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:37 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:38 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:38 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:38 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:42 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:43 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:43 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:43 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:47 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:48 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:48 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:49 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:52 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:53 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:53 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:54 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 04:59:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:57 mds1 multipathd: 8:32: reinstated
Jan 16 04:59:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 04:59:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 04:59:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 04:59:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 04:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 04:59:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 04:59:58 mds1 multipathd: 8:80: reinstated
Jan 16 04:59:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 04:59:58 mds1 multipathd: 8:32: mark as failed
Jan 16 04:59:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 04:59:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 04:59:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 04:59:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 04:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 04:59:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 04:59:59 mds1 multipathd: 8:80: mark as failed
Jan 16 04:59:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 04:59:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:00:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:00:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:00:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:00:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:00:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:00:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:00:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:00:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:00:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:00:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:00:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:00:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:00:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:00:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:00:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:00:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:00:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:00:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:01:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:01:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:01:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:01:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:01:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:01:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:01:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:01:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:01:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:01:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:01:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:01:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:01:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:01:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:01:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:01:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:01:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:01:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:01:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:33 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:53 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:02:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:02:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:02:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:02:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:02:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:02:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:02:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:02:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:02:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:02:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:02:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:02:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:02:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:02:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:02:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:02:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:02:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:02:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:02:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:03:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:03:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:03:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:03:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:03:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:03:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:03:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:03:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:03:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:03:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:03:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:03:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:03:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:03:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:03:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:03:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:03:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:03:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:03:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:04:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:04:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:04:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:04:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:04:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:04:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:04:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:04:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:04:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:04:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:04:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:04:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:04:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:04:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:04:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:04:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:04:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:04:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:04:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:05:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:05:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:05:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:05:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:05:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:05:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:05:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:05:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:05:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:05:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:05:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:05:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:05:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:05:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:05:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:05:58 mds1 multipathd: 8:80: mark as failed
Jan 16 05:05:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:05:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:05:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:06:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:06:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:06:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:06:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:06:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:06:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:06:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:06:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:06:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:06:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:06:58 mds1 multipathd: 8:80: mark as failed
Jan 16 05:06:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:06:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:06:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:06:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:06:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:06:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:06:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:06:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:07:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:07:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:07:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:07:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:07:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:07:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:07:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:07:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:07:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:07:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:07:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:07:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:07:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:07:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:07:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:07:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:07:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:07:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:07:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:53 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:08:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:08:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:08:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:08:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:08:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:08:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:08:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:08:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:08:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:08:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:08:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:08:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:08:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:08:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:08:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:08:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:08:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:08:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:08:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:33 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:53 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:09:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:09:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:09:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:09:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:09:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:09:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:09:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:09:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:09:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:09:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:09:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:09:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:09:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:09:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:09:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:09:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:09:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:09:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:08 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:53 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:10:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:10:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:10:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:10:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:10:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:10:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:10:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:10:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:10:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:10:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:10:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:10:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:10:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:10:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:10:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:10:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:10:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:10:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:18 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:11:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:11:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:11:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:11:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:11:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:11:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:11:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:11:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:11:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:11:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:11:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:11:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:11:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:11:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:11:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:11:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:11:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:11:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:18 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:12:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:12:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:12:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:12:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:12:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:12:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:12:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:12:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:12:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:12:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:12:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:12:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:12:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:12:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:12:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:12:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:12:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:12:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:18 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:38 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:53 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:13:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:13:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:13:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:13:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:13:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:13:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:13:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:13:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:13:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:13:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:13:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:13:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:13:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:13:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:13:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:13:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:13:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:13:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:12 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:13 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:13 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:13 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:17 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:18 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:18 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:22 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:23 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:23 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:23 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:27 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:28 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:28 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:28 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:32 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:33 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:33 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:33 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:37 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:38 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:38 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:42 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:43 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:43 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:43 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:47 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:48 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:48 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:48 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:52 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:53 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:53 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:14:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:57 mds1 multipathd: 8:32: reinstated
Jan 16 05:14:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:14:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:14:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:14:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:14:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:14:58 mds1 multipathd: 8:80: reinstated
Jan 16 05:14:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:14:58 mds1 multipathd: 8:32: mark as failed
Jan 16 05:14:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:14:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:14:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:14:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:14:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:14:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:14:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:14:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:14:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:02 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:03 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:03 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:03 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:07 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:08 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:15:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:15:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:15:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:15:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:15:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:15:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:15:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:15:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:15:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:15:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:15:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:15:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:15:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:15:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:15:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:15:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:15:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:15:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:08 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:16:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:16:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:16:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:16:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:16:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:16:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:16:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:16:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:16:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:16:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:16:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:16:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:16:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:16:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:16:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:16:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:16:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:16:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:17:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:17:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:17:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:17:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:17:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:17:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:17:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:17:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:17:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:17:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:17:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:17:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:17:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:17:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:17:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:17:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:17:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:17:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:18:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:18:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:18:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:18:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:18:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:18:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:18:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:18:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:18:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:18:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:18:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:18:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:18:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:18:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:18:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:18:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:18:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:18:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:19:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:19:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:19:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:19:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:19:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:19:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:19:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:19:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:19:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:19:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:19:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:19:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:19:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:19:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:19:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:19:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:19:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:19:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:20:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:20:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:20:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:20:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:20:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:20:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:20:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:20:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:20:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:20:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:20:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:20:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:20:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:20:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:20:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:20:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:20:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:20:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:21:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:21:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:21:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:21:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:21:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:21:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:21:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:21:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:21:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:21:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:21:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:21:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:21:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:21:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:21:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:21:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:21:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:21:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:22:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:22:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:22:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:22:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:22:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:22:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:22:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:22:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:22:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:22:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:22:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:22:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:22:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:22:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:22:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:22:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:22:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:22:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:23:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:23:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:23:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:23:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:23:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:23:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:23:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:23:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:23:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:23:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:23:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:23:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:23:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:23:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:23:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:23:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:23:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:23:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:24:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:24:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:24:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:24:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:24:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:24:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:24:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:24:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:24:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:24:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:24:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:24:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:24:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:24:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:24:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:24:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:24:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:25:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:25:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:25:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:25:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:25:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:25:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:25:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:25:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:25:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:25:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:25:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:25:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:25:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:25:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:25:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:25:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:25:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:26:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:26:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:26:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:26:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:26:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:26:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:26:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:26:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:26:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:26:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:26:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:26:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:26:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:27:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:27:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:27:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:27:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:27:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:27:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:27:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:27:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:27:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:27:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:27:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:27:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:27:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:27:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:28:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:28:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:28:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:28:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:28:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:28:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:28:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:28:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:28:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:28:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:28:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:29:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:29:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:29:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:29:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:29:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:29:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:29:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:29:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:29:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:29:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:29:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:29:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:29:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:29:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:30:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:30:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:30:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:30:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:30:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:30:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:30:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:30:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:30:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:30:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:30:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:30:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:30:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:30:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:31:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:31:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:31:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:31:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:31:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:31:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:31:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:31:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:31:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:31:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:31:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:31:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:31:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:31:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:32:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:32:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:32:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:32:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:32:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:32:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:32:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:32:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:32:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:32:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:32:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:32:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:32:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:32:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:33:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:33:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:33:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:33:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:33:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:33:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:33:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:33:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:33:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:33:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:33:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:33:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:33:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:33:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:34:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:34:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:34:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:34:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:34:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:34:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:34:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:34:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:34:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:34:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:34:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:34:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:34:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:34:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:35:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:35:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:35:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:35:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:35:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:35:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:35:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:35:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:35:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:35:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:35:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:35:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:35:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:35:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:35:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:35:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:35:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:36:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:36:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:36:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:36:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:36:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:36:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:36:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:36:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:36:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:36:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:36:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:36:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:36:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:36:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:36:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:36:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:36:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:37:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:37:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:37:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:37:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:37:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:37:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:37:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:37:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:37:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:37:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:37:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:37:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:37:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:37:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:37:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:37:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:37:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:38:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:38:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:38:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:38:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:38:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:38:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:38:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:38:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:38:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:38:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:38:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:38:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:38:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:38:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:38:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:38:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:38:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:39:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:39:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:39:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:39:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:39:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:39:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:39:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:39:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:39:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:39:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:39:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:39:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:39:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:39:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:39:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:39:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:39:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:40:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:40:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:40:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:40:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:40:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:40:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:40:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:40:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:40:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:40:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:40:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:40:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:40:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:40:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:40:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:40:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:40:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:41:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:41:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:41:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:41:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:41:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:41:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:41:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:41:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:41:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:41:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:41:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:41:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:41:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:41:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:41:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:41:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:41:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:42:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:42:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:42:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:42:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:42:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:42:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:42:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:42:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:42:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:42:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:42:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:42:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:42:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:42:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:42:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:42:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:42:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:43:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:43:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:43:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:43:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:43:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:43:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:43:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:43:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:43:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:43:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:43:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:43:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:43:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:43:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:43:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:43:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:43:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:44:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:44:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:44:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:44:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:44:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:44:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:44:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:44:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:44:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:44:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:44:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:44:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:44:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:44:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:44:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:44:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:44:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:45:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:45:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:45:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:45:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:45:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:45:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:45:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:45:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:45:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:45:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:45:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:45:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:45:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:45:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:45:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:45:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:45:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:03 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:04 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:04 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:04 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:08 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:09 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:09 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:13 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:14 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:14 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:14 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:18 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:19 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:19 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:19 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:23 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:24 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:24 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:24 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:28 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:29 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:29 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:29 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:33 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:34 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:34 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:34 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:38 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:39 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:39 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:39 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:43 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:44 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:44 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:44 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:48 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:49 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:49 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:49 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:53 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:54 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:54 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:54 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:46:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:58 mds1 multipathd: 8:32: reinstated
Jan 16 05:46:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:46:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:46:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:46:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:46:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:46:59 mds1 multipathd: 8:80: reinstated
Jan 16 05:46:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:46:59 mds1 multipathd: 8:32: mark as failed
Jan 16 05:46:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:46:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:46:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:46:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:46:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:46:59 mds1 multipathd: 8:80: mark as failed
Jan 16 05:46:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:46:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:16 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:47:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:47:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:47:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:47:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:47:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:47:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:47:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:47:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:47:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:47:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:47:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:47:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:47:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:47:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:47:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:47:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:47:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:21 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:51 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:48:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:48:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:48:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:48:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:48:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:48:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:48:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:48:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:48:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:48:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:48:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:48:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:48:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:48:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:48:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:48:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:48:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:21 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:49:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:49:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:49:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:49:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:49:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:49:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:49:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:49:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:49:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:49:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:49:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:49:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:49:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:49:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:49:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:49:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:49:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:49:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:01 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:31 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:50:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:50:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:50:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:50:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:50:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:50:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:50:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:50:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:50:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:50:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:50:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:50:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:50:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:50:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:50:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:50:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:50:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:51:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:51:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:51:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:51:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:55 mds1 multipathd: 8:80: mark as failed
Jan 16 05:51:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:51:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:51:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:51:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:51:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:51:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:51:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:51:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:51:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:51:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:51:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:51:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:46 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:52:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:52:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:52:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:52:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:52:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:52:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:52:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:52:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:52:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:52:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:52:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:52:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:52:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:52:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:52:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:11 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:16 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:21 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:31 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:41 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:46 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:53:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:53:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:53:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:53:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:53:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:53:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:53:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:53:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:53:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:53:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:53:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:53:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:53:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:53:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:53:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:53:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:53:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:11 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:31 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:54:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:54:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:54:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:54:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:54:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:54:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:54:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:54:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:54:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:54:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:54:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:54:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:54:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:54:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:54:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:54:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:54:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:25 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:55:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:55:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:55:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:55:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:55:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:55:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:55:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:55:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:55:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:55:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:55:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:55:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:55:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:55:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:55:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:55:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:55:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:46 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:50 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:56:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:56:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:56:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:56:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:56:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:56:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:56:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:56:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:56:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:56:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:56:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:56:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:56:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:56:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:56:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:56:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:56:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:00 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:05 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:09 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:20 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:31 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:35 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:40 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:45 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:51 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:57:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:57:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:57:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:57:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:57:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:57:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:57:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:57:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:57:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:57:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:57:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:57:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:57:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:57:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:57:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:57:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:57:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:01 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:10 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:15 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:21 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:31 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:41 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:46 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:51 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:58:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:58:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:58:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:58:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:58:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:58:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:58:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:58:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:58:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:58:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:58:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:58:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:58:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:58:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:58:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:58:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:58:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:00 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:00 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:01 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:04 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:05 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:05 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:06 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:09 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:10 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:10 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:11 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:14 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:15 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:15 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:16 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:19 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:20 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:20 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:21 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:24 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:25 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:25 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:26 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:29 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:30 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:30 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:30 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:34 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:35 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:35 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:36 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:39 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:40 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:40 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:41 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:44 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:45 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:45 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:46 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:49 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:50 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:50 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:51 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:54 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 05:59:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:55 mds1 multipathd: 8:80: reinstated
Jan 16 05:59:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 05:59:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:55 mds1 multipathd: 8:32: mark as failed
Jan 16 05:59:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 05:59:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 05:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 05:59:56 mds1 multipathd: 8:80: mark as failed
Jan 16 05:59:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 05:59:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 05:59:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 05:59:59 mds1 multipathd: 8:32: reinstated
Jan 16 05:59:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 05:59:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 05:59:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 05:59:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 05:59:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 05:59:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 05:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 05:59:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:00:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:00:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:00:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:00:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:00:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:00:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:00:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:00:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:00:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:00:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:00:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:00:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:00:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:00:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:00:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:30 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:01:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:01:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:01:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:55 mds1 multipathd: 8:80: mark as failed
Jan 16 06:01:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:01:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:01:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:01:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:01:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:01:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:01:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:01:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:01:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:01:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:02:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:02:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:02:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:02:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:02:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:02:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:02:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:02:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:02:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:02:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:02:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:02:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:02:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:03:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:03:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:03:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:03:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:03:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:03:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:03:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:03:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:03:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:03:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:03:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:03:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:03:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:04:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:04:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:04:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:04:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:04:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:04:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:04:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:04:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:04:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:04:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:04:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:04:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:04:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:04:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:35 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:05:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:05:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:05:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:05:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:05:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:05:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:05:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:05:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:05:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:05:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:05:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:05:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:05:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:05:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:30 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:06:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:06:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:06:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:06:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:06:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:06:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:06:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:06:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:06:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:06:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:06:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:06:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:06:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:35 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:07:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:07:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:07:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:07:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:07:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:07:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:07:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:07:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:07:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:07:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:07:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:07:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:07:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:07:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:07:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:07:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:07:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:08:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:08:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:08:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:08:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:55 mds1 multipathd: 8:80: mark as failed
Jan 16 06:08:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:08:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:08:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:08:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:08:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:08:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:08:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:08:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:08:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:08:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:08:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:08:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:08:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:09:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:09:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:09:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:09:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:09:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:09:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:09:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:09:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:09:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:09:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:09:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:09:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:09:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:09:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:09:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:09:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:09:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:09:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:35 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:10:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:10:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:10:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:10:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:10:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:10:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:10:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:10:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:10:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:10:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:10:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:10:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:10:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:10:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:10:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:10:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:10:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:10:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:11:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:11:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:11:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:11:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:55 mds1 multipathd: 8:80: mark as failed
Jan 16 06:11:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:11:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:11:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:11:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:11:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:11:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:11:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:11:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:11:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:11:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:11:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:11:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:11:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:11:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:12:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:12:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:12:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:12:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:55 mds1 multipathd: 8:80: mark as failed
Jan 16 06:12:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:12:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:12:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:12:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:12:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:12:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:12:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:12:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:12:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:12:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:12:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:12:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:12:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:12:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:13:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:13:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:13:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:13:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:13:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:13:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:13:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:13:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:13:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:13:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:13:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:13:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:13:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:13:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:13:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:13:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:13:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:13:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:30 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:35 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:14:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:14:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:14:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:14:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:14:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:14:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:14:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:14:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:14:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:14:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:14:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:14:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:14:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:14:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:14:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:14:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:14:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:14:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:30 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:15:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:15:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:15:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:15:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:15:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:15:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:15:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:15:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:15:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:15:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:15:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:15:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:15:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:15:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:15:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:15:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:15:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:15:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:30 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:35 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:16:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:16:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:16:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:16:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:16:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:16:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:16:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:16:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:16:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:16:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:16:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:16:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:16:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:16:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:16:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:16:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:16:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:16:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:20 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:30 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:30 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:34 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:35 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:35 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:39 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:40 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:40 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:44 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:45 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:45 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:45 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:49 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:50 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:50 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:50 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:54 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:17:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:55 mds1 multipathd: 8:80: reinstated
Jan 16 06:17:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:17:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:17:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:17:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:55 mds1 multipathd: 8:32: mark as failed
Jan 16 06:17:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:55 mds1 multipathd: 8:80: mark as failed
Jan 16 06:17:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:17:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:17:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:17:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:17:59 mds1 multipathd: 8:32: reinstated
Jan 16 06:17:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:17:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:17:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:17:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:17:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:17:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:17:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:00 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:00 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:00 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:04 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:05 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:05 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:05 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:09 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:10 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:10 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:10 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:14 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:15 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:15 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:15 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:19 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:20 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:20 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:24 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:25 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:25 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:25 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:29 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:18:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:18:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:18:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:18:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:18:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:18:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:18:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:18:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:18:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:18:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:18:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:18:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:18:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:18:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:18:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:18:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:18:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:18:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:19:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:19:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:19:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:19:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:19:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:19:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:19:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:19:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:19:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:19:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:19:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:19:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:19:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:19:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:19:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:19:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:19:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:19:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:20:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:20:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:20:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:20:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:20:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:20:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:20:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:20:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:20:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:20:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:20:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:20:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:20:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:20:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:20:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:20:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:20:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:20:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:21:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:21:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:21:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:21:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:21:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:21:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:21:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:21:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:21:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:21:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:21:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:21:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:21:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:21:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:21:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:21:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:21:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:21:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:22:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:22:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:22:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:22:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:22:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:22:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:22:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:22:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:22:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:22:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:22:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:22:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:22:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:22:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:22:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:22:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:22:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:22:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:23:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:23:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:23:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:23:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:23:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:23:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:23:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:23:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:23:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:23:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:23:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:23:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:23:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:23:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:23:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:23:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:23:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:23:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:24:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:24:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:24:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:24:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:24:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:24:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:24:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:24:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:24:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:24:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:24:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:24:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:24:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:24:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:24:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:24:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:24:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:24:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:25:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:25:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:25:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:25:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:25:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:25:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:25:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:25:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:25:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:25:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:25:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:25:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:25:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:25:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:25:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:25:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:25:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:25:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:26:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:26:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:26:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:26:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:26:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:26:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:26:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:26:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:26:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:26:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:26:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:26:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:26:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:26:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:26:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:26:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:26:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:26:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:26:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:27:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:27:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:27:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:27:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:27:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:27:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:27:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:27:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:27:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:27:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:27:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:27:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:27:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:27:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:27:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:27:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:27:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:27:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:27:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:28:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:28:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:28:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:28:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:28:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:28:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:28:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:28:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:28:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:28:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:28:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:28:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:28:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:28:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:28:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:28:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:28:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:28:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:28:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:29:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:29:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:29:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:29:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:29:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:29:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:29:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:29:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:29:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:29:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:29:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:29:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:29:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:29:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:29:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:29:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:29:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:29:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:29:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:30:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:30:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:30:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:30:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:30:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:30:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:30:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:30:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:30:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:30:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:30:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:30:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:30:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:30:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:30:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:30:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:30:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:30:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:30:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:31:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:31:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:31:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:31:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:31:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:31:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:31:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:31:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:31:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:31:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:31:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:31:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:31:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:31:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:31:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:31:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:31:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:31:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:31:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:32:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:32:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:32:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:32:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:32:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:32:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:32:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:32:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:32:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:32:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:32:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:32:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:32:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:32:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:32:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:32:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:32:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:32:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:32:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:33:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:33:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:33:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:33:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:33:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:33:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:33:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:33:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:33:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:33:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:33:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:33:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:33:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:33:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:33:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:33:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:33:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:33:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:33:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:34:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:34:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:34:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:34:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:34:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:34:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:34:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:34:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:34:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:34:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:34:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:34:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:34:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:34:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:34:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:34:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:34:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:34:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:34:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:35:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:35:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:35:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:35:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:35:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:35:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:35:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:35:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:35:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:35:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:35:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:35:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:35:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:35:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:35:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:35:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:35:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:36:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:36:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:36:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:36:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:36:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:36:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:36:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:36:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:36:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:36:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:36:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:36:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:36:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:36:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:36:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:36:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:36:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:37:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:37:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:37:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:37:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:37:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:37:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:37:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:37:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:37:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:37:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:37:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:37:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:37:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:37:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:37:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:37:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:37:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:38:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:38:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:38:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:38:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:38:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:38:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:38:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:38:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:38:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:38:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:38:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:38:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:38:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:38:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:38:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:38:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:38:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:39:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:39:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:39:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:39:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:39:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:39:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:39:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:39:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:39:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:39:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:39:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:39:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:39:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:39:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:39:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:39:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:39:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:06 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:40 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:40:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:40:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:40:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:40:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:40:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:40:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:40:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:40:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:40:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:40:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:40:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:40:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:40:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:40:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:40:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:40:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:40:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:41:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:41:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:41:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:41:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:41:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:41:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:41:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:41:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:41:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:41:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:41:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:41:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:41:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:41:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:41:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:41:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:41:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:42:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:42:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:42:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:42:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:42:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:42:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:42:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:42:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:42:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:42:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:42:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:42:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:42:56 mds1 multipathd: 8:80: mark as failed
Jan 16 06:42:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:42:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:42:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:42:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:42:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:21 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:31 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:36 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:41 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:51 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:43:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:43:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:43:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:43:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:43:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:43:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:43:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:43:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:43:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:43:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:43:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:43:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:43:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:43:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:43:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:43:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:43:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:43:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:44:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:44:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:44:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:44:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:44:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:44:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:44:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:44:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:44:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:44:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:44:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:44:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:44:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:44:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:44:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:44:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:44:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:44:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:45:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:45:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:45:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:45:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:45:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:45:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:45:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:45:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:45:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:45:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:45:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:45:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:45:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:45:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:45:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:45:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:45:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:45:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:26 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:46:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:46:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:46:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:46:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:46:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:46:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:46:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:46:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:46:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:46:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:46:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:46:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:46:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:46:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:46:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:46:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:46:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:46:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:47:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:47:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:47:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:47:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:47:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:47:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:47:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:47:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:47:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:47:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:47:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:47:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:47:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:47:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:47:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:47:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:47:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:47:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:48:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:48:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:48:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:48:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:48:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:48:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:48:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:48:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:48:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:48:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:48:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:48:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:48:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:48:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:48:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:48:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:48:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:48:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:16 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:49:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:49:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:49:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:49:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:49:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:49:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:49:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:49:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:49:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:49:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:49:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:49:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:49:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:49:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:49:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:49:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:49:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:49:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:50:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:50:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:50:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:50:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:50:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:50:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:50:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:50:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:50:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:50:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:50:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:50:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:50:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:50:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:50:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:50:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:50:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:50:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:05 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:06 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:06 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:10 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:11 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:11 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:11 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:15 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:16 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:16 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:20 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:21 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:21 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:25 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:26 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:26 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:30 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:31 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:31 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:32 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:35 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:36 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:40 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:41 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:41 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:42 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:45 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:46 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:46 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:46 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:50 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:51 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:51 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:51:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:55 mds1 multipathd: 8:32: reinstated
Jan 16 06:51:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:51:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:51:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:51:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:51:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:51:56 mds1 multipathd: 8:80: reinstated
Jan 16 06:51:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:51:56 mds1 multipathd: 8:32: mark as failed
Jan 16 06:51:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:51:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:51:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:51:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:51:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:51:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:51:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:51:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:51:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:00 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:01 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:01 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:01 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:53 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:52:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:52:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:52:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:52:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:52:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:52:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:52:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:52:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:52:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:52:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:52:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:52:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:52:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:52:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:52:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:52:58 mds1 multipathd: 8:80: mark as failed
Jan 16 06:52:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:52:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:03 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:08 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:53 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:53:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:53:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:53:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:53:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:53:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:53:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:53:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:53:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:53:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:53:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:53:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:53:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:53:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:53:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:53:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:53:58 mds1 multipathd: 8:80: mark as failed
Jan 16 06:53:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:53:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:03 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:08 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:18 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:48 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:53 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:54:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:54:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:54:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:54:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:54:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:54:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:54:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:54:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:54:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:54:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:54:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:54:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:54:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:54:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:54:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:54:58 mds1 multipathd: 8:80: mark as failed
Jan 16 06:54:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:54:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:03 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:08 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:18 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:55:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:55:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:55:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:55:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:55:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:55:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:55:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:55:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:55:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:55:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:55:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:55:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:55:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:55:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:55:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:55:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:55:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:55:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:12 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:18 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:36 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:48 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:53 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:56:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:56:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:56:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:56:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:56:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:56:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:56:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:56:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:56:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:56:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:56:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:56:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:56:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:56:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:56:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:56:58 mds1 multipathd: 8:80: mark as failed
Jan 16 06:56:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:56:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:03 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:08 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:22 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:28 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:48 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:53 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:57:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:57:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:57:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:57:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:57:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:57:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:57:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:57:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:57:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:57:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:57:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:57:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:57:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:57:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:57:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:57:58 mds1 multipathd: 8:80: mark as failed
Jan 16 06:57:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:57:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:07 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:38 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:58:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:58:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:58:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:58:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:58:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:58:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:58:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:58:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:58:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:58:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:58:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:58:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:58:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:58:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:58:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:58:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:58:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:58:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:01 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:02 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:02 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:02 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:06 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:07 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:07 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:08 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:11 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:12 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:12 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:13 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:16 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:17 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:17 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:17 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:21 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:22 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:22 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:23 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:26 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:27 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:27 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:27 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:31 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:32 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:32 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:33 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:36 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:37 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:37 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:37 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:41 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:42 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:42 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:43 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:46 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:47 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:47 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:47 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:51 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:52 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:52 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:52 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:56 mds1 multipathd: 8:32: reinstated
Jan 16 06:59:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 06:59:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 06:59:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 06:59:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 06:59:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 06:59:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 06:59:57 mds1 multipathd: 8:80: reinstated
Jan 16 06:59:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 06:59:57 mds1 multipathd: 8:32: mark as failed
Jan 16 06:59:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 06:59:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 06:59:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 06:59:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 06:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 06:59:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 06:59:57 mds1 multipathd: 8:80: mark as failed
Jan 16 06:59:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 06:59:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 06:59:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:00:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:00:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:00:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:00:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:00:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:00:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:00:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:00:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:00:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:00:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:00:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:00:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:00:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:00:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:00:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:00:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:00:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:00:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:01:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:01:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:01:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:01:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:01:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:01:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:01:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:01:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:01:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:01:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:01:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:01:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:01:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:01:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:01:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:01:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:01:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:01:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:01:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:02:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:02:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:02:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:02:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:02:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:02:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:02:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:02:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:02:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:02:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:02:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:02:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:02:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:02:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:02:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:02:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:02:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:02:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:02:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:03:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:03:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:03:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:03:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:03:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:03:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:03:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:03:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:03:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:03:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:03:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:03:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:03:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:03:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:03:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:03:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:03:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:03:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:03:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:04:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:04:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:04:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:04:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:04:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:04:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:04:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:04:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:04:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:04:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:04:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:04:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:04:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:04:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:04:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:04:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:04:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:04:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:04:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:05:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:05:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:05:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:05:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:05:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:05:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:05:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:05:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:05:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:05:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:05:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:05:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:05:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:05:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:05:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:05:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:05:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:05:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:05:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:06:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:06:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:06:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:06:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:06:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:06:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:06:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:06:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:06:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:06:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:06:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:06:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:06:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:06:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:06:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:06:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:06:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:06:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:06:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:07:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:07:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:07:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:07:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:07:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:07:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:07:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:07:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:07:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:07:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:07:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:07:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:07:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:07:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:07:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:07:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:07:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:07:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:08:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:08:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:08:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:08:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:08:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:08:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:08:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:08:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:08:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:08:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:08:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:08:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:08:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:08:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:08:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:08:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:08:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:08:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:08:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:09:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:09:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:09:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:09:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:09:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:09:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:09:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:09:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:09:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:09:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:09:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:09:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:09:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:09:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:09:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:09:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:09:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:09:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:10:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:10:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:10:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:10:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:10:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:10:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:10:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:10:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:10:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:10:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:10:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:10:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:10:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:10:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:10:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:10:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:10:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:10:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:11:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:11:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:11:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:11:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:11:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:11:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:11:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:11:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:11:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:11:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:11:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:11:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:11:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:11:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:11:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:11:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:11:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:11:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:12:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:12:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:12:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:12:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:12:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:12:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:12:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:12:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:12:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:12:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:12:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:12:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:12:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:12:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:12:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:12:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:12:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:12:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:13:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:13:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:13:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:13:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:13:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:13:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:13:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:13:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:13:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:13:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:13:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:13:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:13:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:13:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:13:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:13:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:13:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:13:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:14:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:14:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:14:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:14:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:14:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:14:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:14:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:14:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:14:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:14:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:14:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:14:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:14:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:14:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:14:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:14:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:14:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:14:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:15:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:15:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:15:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:15:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:15:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:15:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:15:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:15:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:15:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:15:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:15:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:15:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:15:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:15:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:15:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:15:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:15:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:15:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:15:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:16:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:16:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:16:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:16:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:16:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:16:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:16:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:16:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:16:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:16:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:16:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:16:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:16:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:16:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:16:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:16:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:16:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:16:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:16:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:17:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:17:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:17:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:17:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:17:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:17:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:17:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:17:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:17:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:17:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:17:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:17:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:17:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:17:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:17:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:17:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:17:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:17:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:17:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:18:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:18:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:18:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:18:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:18:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:18:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:18:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:18:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:18:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:18:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:18:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:18:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:18:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:18:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:18:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:18:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:18:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:18:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:19:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:19:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:19:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:19:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:19:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:19:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:19:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:19:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:19:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:19:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:19:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:19:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:19:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:19:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:19:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:19:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:19:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:19:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:20:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:20:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:20:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:20:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:20:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:20:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:20:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:20:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:20:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:20:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:20:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:20:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:20:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:20:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:20:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:20:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:20:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:20:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:21:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:21:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:21:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:21:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:21:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:21:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:21:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:21:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:21:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:21:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:21:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:21:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:21:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:21:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:21:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:21:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:21:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:21:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:22:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:22:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:22:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:22:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:22:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:22:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:22:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:22:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:22:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:22:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:22:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:22:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:22:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:22:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:22:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:22:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:22:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:22:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:23:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:23:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:23:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:23:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:23:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:23:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:23:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:23:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:23:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:23:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:23:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:23:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:23:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:23:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:23:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:23:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:23:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:23:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:24:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:24:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:24:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:24:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:24:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:24:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:24:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:24:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:24:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:24:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:24:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:24:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:24:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:24:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:24:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:24:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:24:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:06 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:06 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:06 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:06 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:06 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:06 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:06 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:06 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:07 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:07 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:07 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:07 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:07 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:07 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:07 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:07 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:07 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:07 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:07 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:11 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:11 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:11 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:11 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:11 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:11 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:11 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:11 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:12 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:12 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:12 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:12 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:12 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:12 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:12 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:12 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:12 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:12 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:12 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:16 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:16 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:16 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:16 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:16 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:16 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:16 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:16 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:17 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:17 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:17 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:17 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:17 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:17 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:17 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:17 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:17 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:17 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:17 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:21 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:21 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:21 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:21 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:21 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:21 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:21 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:21 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:22 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:22 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:22 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:22 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:22 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:22 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:22 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:22 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:22 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:22 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:22 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:26 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:26 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:26 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:26 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:26 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:26 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:26 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:26 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:27 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:27 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:27 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:27 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:27 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:27 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:27 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:27 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:27 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:27 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:27 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:31 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:31 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:31 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:31 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:31 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:31 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:31 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:31 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:32 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:32 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:32 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:32 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:32 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:32 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:32 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:32 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:32 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:36 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:36 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:36 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:36 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:36 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:36 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:36 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:36 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:37 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:37 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:37 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:37 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:37 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:37 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:37 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:37 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:37 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:37 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:37 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:41 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:41 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:41 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:41 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:41 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:41 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:41 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:41 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:42 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:42 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:42 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:42 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:42 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:42 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:42 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:42 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:42 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:42 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:46 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:46 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:46 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:46 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:46 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:46 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:46 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:46 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:47 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:47 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:47 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:47 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:47 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:47 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:47 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:47 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:47 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:47 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:47 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:51 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:51 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:51 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:51 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:51 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:51 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:51 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:51 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:52 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:52 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:52 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:52 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:52 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:52 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:52 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:52 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:52 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:52 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:52 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:56 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:56 mds1 multipathd: 8:32: reinstated
Jan 16 07:25:56 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:25:56 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:25:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:56 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:25:56 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:56 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:56 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:25:57 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:25:57 mds1 multipathd: 8:80: reinstated
Jan 16 07:25:57 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:25:57 mds1 multipathd: 8:32: mark as failed
Jan 16 07:25:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:25:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:57 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:25:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:25:57 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:25:57 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:25:57 mds1 multipathd: 8:80: mark as failed
Jan 16 07:25:57 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:25:57 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:25:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:01 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:01 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:01 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:01 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:01 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:01 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:01 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:01 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:02 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:02 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:02 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:02 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:02 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:02 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:02 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:02 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:26:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:26:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:26:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:26:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:26:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:26:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:26:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:26:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:26:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:26:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:26:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:26:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:26:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:26:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:26:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:26:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:26:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:26:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:27:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:27:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:27:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:27:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:27:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:27:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:27:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:27:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:27:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:27:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:27:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:27:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:27:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:27:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:27:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:27:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:27:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:27:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:28:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:28:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:28:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:28:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:28:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:28:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:28:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:28:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:28:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:28:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:28:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:28:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:28:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:28:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:28:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:28:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:28:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:29:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:29:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:29:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:29:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:29:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:29:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:29:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:29:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:29:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:29:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:29:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:29:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:29:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:29:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:29:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:29:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:29:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:29:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:30:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:30:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:30:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:30:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:30:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:30:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:30:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:30:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:30:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:30:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:30:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:30:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:30:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:30:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:30:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:30:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:30:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:30:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:31:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:31:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:31:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:31:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:31:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:31:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:31:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:31:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:31:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:31:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:31:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:31:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:31:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:31:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:31:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:31:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:31:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:31:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:32:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:32:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:32:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:32:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:32:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:32:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:32:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:32:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:32:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:32:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:32:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:32:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:32:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:32:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:32:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:32:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:32:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:32:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:33:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:33:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:33:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:33:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:33:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:33:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:33:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:33:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:33:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:33:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:33:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:33:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:33:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:33:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:33:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:33:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:33:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:33:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:34:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:34:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:34:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:34:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:34:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:34:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:34:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:34:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:34:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:34:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:34:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:34:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:34:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:34:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:34:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:34:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:34:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:34:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:35:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:35:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:35:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:35:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:35:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:35:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:35:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:35:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:35:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:35:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:35:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:35:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:35:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:35:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:35:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:35:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:35:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:35:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:36:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:36:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:36:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:36:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:36:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:36:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:36:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:36:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:36:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:36:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:36:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:36:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:36:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:36:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:36:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:36:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:36:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:36:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:37:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:37:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:37:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:37:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:37:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:37:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:37:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:37:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:37:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:37:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:37:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:37:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:37:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:37:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:37:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:37:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:37:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:37:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:38:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:38:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:38:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:38:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:38:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:38:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:38:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:38:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:38:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:38:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:38:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:38:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:38:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:38:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:38:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:38:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:38:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:38:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:39:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:39:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:39:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:39:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:39:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:39:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:39:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:39:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:39:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:39:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:39:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:39:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:39:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:39:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:39:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:39:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:39:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:39:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:40:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:40:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:40:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:40:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:40:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:40:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:40:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:40:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:40:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:40:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:40:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:40:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:40:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:40:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:40:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:40:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:40:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:40:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:41:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:41:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:41:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:41:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:41:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:41:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:41:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:41:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:41:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:41:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:41:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:41:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:41:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:41:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:41:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:41:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:41:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:41:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:42:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:42:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:42:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:42:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:42:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:42:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:42:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:42:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:42:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:42:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:42:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:42:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:42:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:42:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:42:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:42:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:42:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:42:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:43:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:43:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:43:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:43:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:43:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:43:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:43:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:43:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:43:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:43:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:43:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:43:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:43:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:43:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:43:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:43:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:43:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:43:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:28 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:28 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:28 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:44:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:44:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:44:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:44:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:44:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:44:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:44:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:44:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:44:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:44:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:44:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:44:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:44:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:44:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:44:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:44:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:44:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:44:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:38 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:38 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:38 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:45:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:45:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:45:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:45:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:45:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:45:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:45:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:45:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:45:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:45:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:45:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:45:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:45:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:45:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:45:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:45:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:45:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:45:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:23 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:23 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:23 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:46:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:46:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:46:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:46:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:46:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:46:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:46:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:46:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:46:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:46:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:46:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:46:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:46:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:46:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:46:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:46:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:46:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:46:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:47:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:47:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:47:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:47:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:47:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:47:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:47:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:47:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:47:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:47:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:47:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:47:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:47:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:47:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:47:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:47:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:47:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:47:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:08 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:08 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:08 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:53 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:53 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:53 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:48:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:48:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:48:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:48:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:48:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:48:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:48:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:48:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:48:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:48:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:48:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:48:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:48:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:48:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:48:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:48:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:48:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:48:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:49:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:49:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:49:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:49:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:49:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:49:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:49:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:49:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:49:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:49:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:49:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:49:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:49:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:49:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:49:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:49:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:49:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:49:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:50:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:50:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:50:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:50:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:50:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:50:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:50:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:50:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:50:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:50:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:50:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:50:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:50:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:50:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:50:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:50:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:50:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:50:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:51:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:51:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:51:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:51:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:51:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:51:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:51:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:51:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:51:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:51:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:51:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:51:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:51:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:51:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:51:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:51:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:51:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:51:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:03 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:03 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:03 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:13 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:13 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:13 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:52:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:52:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:52:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:52:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:52:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:52:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:52:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:52:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:52:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:52:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:52:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:52:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:52:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:52:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:52:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:52:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:52:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:43 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:43 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:43 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:53:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:53:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:53:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:53:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:53:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:53:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:53:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:53:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:53:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:53:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:53:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:53:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:53:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:53:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:53:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:53:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:53:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:54:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:54:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:54:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:54:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:54:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:54:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:54:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:54:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:54:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:54:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:54:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:54:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:54:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:54:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:54:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:54:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:54:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:55:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:55:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:55:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:55:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:55:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:55:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:55:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:55:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:55:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:55:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:55:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:55:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:55:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:55:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:55:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:55:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:55:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:56:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:56:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:56:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:56:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:56:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:56:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:56:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:56:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:56:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:56:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:56:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:56:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:56:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:56:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:56:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:56:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:56:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:57:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:57:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:57:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:57:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:57:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:57:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:57:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:57:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:57:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:57:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:57:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:57:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:57:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:57:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:57:59 mds1 multipathd: 8:80: mark as failed
Jan 16 07:57:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:57:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:18 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:18 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:18 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:34 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:48 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:48 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:48 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:58:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:58:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:58:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:58:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:58:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:58:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:58:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:58:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:58:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:58:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:58:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:58:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:58:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:58:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:58:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:58:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:58:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:02 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:03 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:03 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:04 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:07 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:08 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:08 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:09 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:12 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:13 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:13 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:14 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:17 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:18 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:18 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:19 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:22 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:23 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:23 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:24 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:27 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:28 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:28 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:29 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:32 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:33 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:33 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:33 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:33 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:33 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:37 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:38 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:38 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:39 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:42 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:43 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:43 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:44 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:47 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:48 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:48 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:49 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:52 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:53 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:53 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:54 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:57 mds1 multipathd: 8:32: reinstated
Jan 16 07:59:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 07:59:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 07:59:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 07:59:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 07:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 07:59:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 07:59:58 mds1 multipathd: 8:80: reinstated
Jan 16 07:59:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 07:59:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:58 mds1 multipathd: 8:32: mark as failed
Jan 16 07:59:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 07:59:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 07:59:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 07:59:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 07:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 07:59:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 07:59:58 mds1 multipathd: 8:80: mark as failed
Jan 16 07:59:58 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 07:59:58 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 07:59:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:02 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:03 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:03 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:07 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:07 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:07 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:07 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:07 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:07 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:07 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:07 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:07 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:07 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:08 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:08 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:08 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:08 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:08 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:08 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:08 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:08 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:12 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:12 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:12 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:12 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:12 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:12 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:12 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:12 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:12 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:12 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:13 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:13 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:13 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:13 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:13 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:13 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:13 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:13 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:17 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:17 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:17 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:17 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:17 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:17 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:17 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:17 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:17 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:17 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:18 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:18 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:18 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:18 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:18 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:18 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:18 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:18 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:22 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:22 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:22 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:22 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:22 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:22 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:22 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:22 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:22 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:22 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:23 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:23 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:23 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:23 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:23 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:23 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:23 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:23 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:27 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:27 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:27 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:27 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:27 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:27 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:27 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:27 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:27 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:27 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:28 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:28 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:28 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:28 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:28 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:28 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:28 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:28 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:32 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:32 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:32 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:32 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:32 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:32 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:32 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:32 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:32 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:32 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:33 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:33 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:33 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:33 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:33 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:33 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:33 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:33 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:37 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:37 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:37 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:37 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:37 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:37 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:37 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:37 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:37 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:37 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:38 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:38 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:38 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:38 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:38 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:38 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:38 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:38 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:39 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:42 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:42 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:42 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:42 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:42 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:42 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:42 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:42 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:42 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:42 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:43 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:43 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:43 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:43 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:43 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:43 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:43 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:43 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:47 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:47 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:47 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:47 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:47 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:47 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:47 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:47 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:47 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:47 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:48 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:48 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:48 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:48 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:48 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:48 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:48 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:48 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:52 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:52 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:52 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:52 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:52 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:52 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:52 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:52 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:52 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:52 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:53 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:53 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:53 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:53 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:53 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:53 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:53 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:53 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:00:57 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:57 mds1 multipathd: 8:32: reinstated
Jan 16 08:00:57 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:00:57 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:00:57 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:57 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:00:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:57 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:57 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:00:58 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:00:58 mds1 multipathd: 8:80: reinstated
Jan 16 08:00:58 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:00:58 mds1 multipathd: 8:32: mark as failed
Jan 16 08:00:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:00:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:58 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:00:58 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:00:58 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:00:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:00:58 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:00:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:00:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:00:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:02 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:02 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:02 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:02 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:02 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:02 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:02 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:03 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:03 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:03 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:03 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:03 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:03 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:03 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:03 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:01:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:01:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:01:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:01:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:01:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:01:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:01:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:01:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:01:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:01:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:01:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:01:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:01:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:01:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:01:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:01:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:01:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:01:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:39 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:02:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:02:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:02:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:02:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:02:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:02:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:02:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:02:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:02:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:02:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:02:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:02:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:02:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:02:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:02:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:02:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:02:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:02:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:03:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:03:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:03:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:03:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:03:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:03:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:03:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:03:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:03:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:03:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:03:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:03:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:03:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:03:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:03:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:03:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:03:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:03:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:39 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:04:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:04:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:04:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:04:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:04:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:04:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:04:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:04:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:04:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:04:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:04:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:04:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:04:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:04:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:04:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:04:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:04:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:04:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:05:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:05:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:05:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:05:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:05:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:05:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:05:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:05:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:05:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:05:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:05:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:05:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:05:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:05:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:05:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:05:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:05:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:05:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:06:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:06:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:06:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:06:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:06:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:06:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:06:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:06:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:06:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:06:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:06:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:06:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:06:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:06:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:06:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:06:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:06:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:07:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:07:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:07:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:07:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:07:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:07:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:07:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:07:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:07:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:07:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:07:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:07:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:07:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:07:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:07:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:07:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:07:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:07:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:08:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:08:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:08:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:08:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:08:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:08:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:08:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:08:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:08:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:08:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:08:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:08:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:08:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:08:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:08:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:08:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:08:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:08:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:09:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:09:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:09:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:09:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:09:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:09:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:09:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:09:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:09:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:09:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:09:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:09:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:09:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:09:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:09:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:09:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:09:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:09:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:10:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:10:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:10:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:10:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:10:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:10:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:10:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:10:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:10:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:10:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:10:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:10:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:10:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:10:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:10:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:10:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:10:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:10:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:11:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:11:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:11:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:11:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:11:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:11:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:11:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:11:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:11:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:11:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:11:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:11:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:11:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:11:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:11:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:11:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:11:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:11:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:12:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:12:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:12:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:12:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:12:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:12:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:12:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:12:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:12:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:12:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:12:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:12:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:12:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:12:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:12:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:12:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:12:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:12:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:13:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:13:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:13:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:13:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:13:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:13:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:13:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:13:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:13:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:13:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:13:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:13:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:13:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:13:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:13:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:13:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:13:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:13:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:14:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:14:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:14:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:14:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:14:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:14:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:14:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:14:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:14:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:14:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:14:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:14:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:14:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:14:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:14:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:14:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:14:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:14:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:15:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:15:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:15:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:15:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:15:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:15:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:15:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:15:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:15:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:15:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:15:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:15:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:15:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:15:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:15:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:15:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:15:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:15:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:16:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:16:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:16:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:16:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:16:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:16:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:16:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:16:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:16:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:16:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:16:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:16:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:16:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:16:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:16:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:16:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:16:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:16:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:17:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:17:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:17:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:17:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:17:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:17:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:17:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:17:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:17:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:17:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:17:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:17:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:17:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:17:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:17:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:17:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:17:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:17:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:18:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:18:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:18:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:18:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:18:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:18:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:18:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:18:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:18:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:18:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:18:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:18:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:18:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:18:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:18:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:18:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:18:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:18:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:19:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:19:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:19:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:19:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:19:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:19:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:19:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:19:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:19:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:19:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:19:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:19:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:19:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:19:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:19:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:19:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:19:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:19:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:20:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:20:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:20:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:20:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:20:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:20:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:20:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:20:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:20:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:20:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:20:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:20:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:20:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:20:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:20:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:20:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:20:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:20:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:21:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:21:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:21:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:21:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:21:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:21:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:21:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:21:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:21:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:21:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:21:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:21:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:21:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:21:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:21:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:21:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:21:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:21:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:22:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:22:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:22:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:22:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:22:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:22:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:22:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:22:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:22:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:22:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:22:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:22:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:22:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:22:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:22:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:22:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:22:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:22:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:14 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:14 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:14 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:23:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:23:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:23:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:23:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:23:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:23:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:23:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:23:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:23:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:23:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:23:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:23:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:23:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:23:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:23:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:23:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:23:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:23:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:24:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:24:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:24:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:24:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:24:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:24:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:24:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:24:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:24:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:24:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:24:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:24:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:24:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:24:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:24:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:24:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:24:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:24 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:24 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:24 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:25:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:25:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:25:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:25:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:25:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:25:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:25:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:25:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:25:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:25:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:25:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:25:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:25:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:25:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:25:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:25:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:25:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:26:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:26:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:26:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:26:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:26:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:26:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:26:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:26:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:26:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:26:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:26:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:26:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:26:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:26:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:26:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:26:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:27:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:27:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:27:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:27:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:27:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:27:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:27:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:27:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:27:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:27:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:27:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:27:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:27:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:27:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:27:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:27:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:28:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:28:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:28:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:28:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:28:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:28:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:28:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:28:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:28:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:28:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:28:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:28:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:28:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:28:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:28:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:29:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:29:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:29:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:29:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:29:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:29:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:29:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:29:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:29:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:29:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:29:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:29:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:29:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:29:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:29:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:29:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:44 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:44 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:44 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:54 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:54 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:54 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:30:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:30:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:30:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:30:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:30:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:30:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:30:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:30:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:30:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:30:59 mds1 multipathd: 8:80: mark as failed
Jan 16 08:30:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:59 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:30:59 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:30:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:30:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:30:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:30:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:31:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:31:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:31:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:31:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:31:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:31:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:31:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:31:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:31:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:31:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:31:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:31:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:31:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:31:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:31:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:31:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:34 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:34 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:34 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:39 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:32:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:32:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:32:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:32:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:32:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:32:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:32:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:32:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:32:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:32:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:32:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:32:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:32:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:32:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:32:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:32:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:09 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:09 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:09 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:39 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:39 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:39 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:33:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:33:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:33:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:33:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:33:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:33:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:33:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:33:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:33:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:33:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:33:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:33:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:33:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:33:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:33:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:33:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:04 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:04 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:04 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:29 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:29 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:29 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:29 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:29 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:29 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:29 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:29 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:33 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:33 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:33 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:33 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:33 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:33 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:33 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:33 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:33 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:33 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:34 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:34 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:34 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:34 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:34 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:34 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:34 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:34 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:38 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:38 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:38 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:38 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:38 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:38 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:38 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:38 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:38 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:38 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:39 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:39 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:39 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:39 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:39 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:39 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:39 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:39 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:43 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:43 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:43 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:43 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:43 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:43 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:43 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:43 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:43 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:43 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:44 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:44 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:44 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:44 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:44 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:44 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:44 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:44 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:48 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:48 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:48 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:48 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:48 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:48 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:48 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:48 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:48 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:48 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:49 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:49 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:49 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:49 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:49 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:49 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:49 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:49 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:49 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:49 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:49 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:53 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:53 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:53 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:53 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:53 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:53 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:53 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:53 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:53 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:53 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:54 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:54 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:54 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:54 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:54 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:54 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:54 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:54 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:34:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:34:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:34:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:34:58 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:58 mds1 multipathd: 8:32: reinstated
Jan 16 08:34:58 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:34:58 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:34:58 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:58 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:34:58 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:58 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:58 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:58 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:34:59 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:34:59 mds1 multipathd: 8:80: reinstated
Jan 16 08:34:59 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:59 mds1 multipathd: 8:32: mark as failed
Jan 16 08:34:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:34:59 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:34:59 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:34:59 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:34:59 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:03 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:03 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:03 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:03 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:03 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:03 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:03 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:03 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:03 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:03 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:04 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:04 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:04 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:04 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:04 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:04 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:04 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:04 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:08 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:08 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:08 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:08 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:08 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:08 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:08 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:08 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:08 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:08 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:09 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:09 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:09 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:09 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:09 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:09 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:09 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:09 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:13 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:13 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:13 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:13 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:13 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:13 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:13 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:13 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:13 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:13 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:14 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:14 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:14 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:14 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:14 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:14 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:14 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:14 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:18 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:18 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:18 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:18 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:18 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:18 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:18 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:18 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:18 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:18 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:19 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:19 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:19 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:19 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:19 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:19 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:19 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:19 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:19 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:19 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:19 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:23 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:23 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:23 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:23 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:23 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:23 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:23 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:23 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:23 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:23 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:24 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:24 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:24 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:24 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:24 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:24 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:24 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:24 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:28 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:28 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:28 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:28 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:28 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:28 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:28 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:28 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:28 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:28 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:29 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:29 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:29 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:35:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:35:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:35:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:35:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:35:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:35:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:35:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:35:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:35:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:35:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:35:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:35:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:35:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:35:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:35:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:35:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:35:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:36:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:36:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:36:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:36:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:36:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:36:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:36:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:36:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:36:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:36:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:36:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:36:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:36:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:36:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:36:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:36:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:36:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:37:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:37:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:37:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:37:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:37:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:37:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:37:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:37:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:37:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:37:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:37:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:37:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:37:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:37:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:37:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:37:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:37:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:38:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:38:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:38:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:38:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:38:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:38:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:38:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:38:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:38:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:38:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:38:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:38:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:38:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:38:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:38:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:38:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:38:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:39:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:39:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:39:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:39:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:39:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:39:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:39:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:39:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:39:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:39:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:39:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:39:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:39:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:39:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:39:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:39:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:39:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:40:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:40:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:40:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:40:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:40:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:40:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:40:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:40:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:40:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:40:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:40:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:40:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:40:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:40:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:40:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:40:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:40:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:41:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:41:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:41:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:41:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:41:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:41:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:41:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:41:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:41:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:41:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:41:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:41:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:41:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:41:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:41:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:41:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:41:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:21 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:35 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:42:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:42:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:42:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:42:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:42:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:42:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:42:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:42:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:42:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:42:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:42:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:42:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:42:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:42:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:42:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:42:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:42:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:43:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:43:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:43:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:43:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:43:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:43:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:43:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:43:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:43:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:43:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:43:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:43:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:43:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:43:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:43:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:43:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:43:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:44:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:44:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:44:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:44:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:44:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:44:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:44:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:44:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:44:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:44:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:44:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:44:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:44:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:44:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:44:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:44:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:44:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:45:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:45:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:45:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:45:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:45:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:45:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:45:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:45:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:45:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:45:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:45:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:45:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:45:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:45:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:45:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:45:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:45:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:21 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:46:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:46:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:46:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:46:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:46:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:46:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:46:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:46:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:46:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:46:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:46:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:46:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:46:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:46:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:46:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:46:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:46:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:47:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:47:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:47:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:47:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:47:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:47:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:47:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:47:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:47:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:47:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:47:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:47:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:47:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:47:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:47:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:47:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:47:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:48:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:48:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:48:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:48:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:48:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:48:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:48:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:48:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:48:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:48:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:48:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:48:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:48:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:48:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:48:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:48:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:48:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:49:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:49:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:49:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:49:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:49:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:49:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:49:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:49:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:49:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:49:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:49:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:49:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:49:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:49:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:49:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:49:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:49:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:50:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:50:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:50:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:50:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:50:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:50:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:50:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:50:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:50:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:50:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:50:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:50:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:50:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:50:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:50:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:50:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:50:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:21 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:51:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:51:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:51:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:51:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:51:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:51:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:51:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:51:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:51:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:51:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:51:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:51:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:51:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:51:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:51:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:51:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:51:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:40 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:52:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:52:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:52:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:52:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:52:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:52:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:52:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:52:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:52:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:52:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:52:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:52:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:52:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:52:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:52:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:52:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:21 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:53:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:53:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:53:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:53:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:53:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:53:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:53:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:53:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:53:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:53:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:53:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:53:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:53:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:53:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:53:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:53:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:53:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:15 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:54:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:54:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:54:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:54:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:54:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:54:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:54:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:54:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:54:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:54:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:54:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:54:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:54:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:54:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:54:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:54:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:54:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:45 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:55:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:55:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:55:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:55:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:55:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:55:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:55:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:55:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:55:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:55:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:55:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:55:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:55:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:55:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:55:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:55:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:55:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:00 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:10 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:25 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:56:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:56:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:56:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:56:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:56:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:56:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:56:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:56:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:56:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:56:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:56:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:56:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:56:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:56:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:56:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:56:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:56:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:30 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:50 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:57:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:57:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:57:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:57:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:57:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:57:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:57:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:57:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:57:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:57:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:57:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:57:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:57:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:57:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:57:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:57:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:57:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:05 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:05 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:05 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:58:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:58:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:58:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:58:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:58:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:58:55 mds1 multipathd: 8:80: mark as failed
Jan 16 08:58:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:58:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:58:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:58:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:58:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:58:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:58:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:58:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:58:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:58:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:58:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:00 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:00 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:01 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:04 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:05 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:05 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:06 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:09 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:10 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:10 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:11 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:14 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:15 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:15 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:16 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:19 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:20 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:20 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:20 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:24 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:25 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:25 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:26 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:29 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:30 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:30 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:31 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:34 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:35 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:35 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:36 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:39 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:40 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:40 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:41 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:44 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:45 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:45 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:46 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:49 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:50 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:50 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:51 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:54 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 08:59:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:55 mds1 multipathd: 8:80: reinstated
Jan 16 08:59:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 08:59:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:55 mds1 multipathd: 8:32: mark as failed
Jan 16 08:59:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 08:59:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 08:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 08:59:56 mds1 multipathd: 8:80: mark as failed
Jan 16 08:59:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 08:59:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 08:59:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 08:59:59 mds1 multipathd: 8:32: reinstated
Jan 16 08:59:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 08:59:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 08:59:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 08:59:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 08:59:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 08:59:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 08:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 08:59:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:01 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:10 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:15 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:26 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:31 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:35 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:41 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:00:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:00:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:00:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:00:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:00:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:00:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:00:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:00:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:00:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:00:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:00:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:00:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:00:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:00:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:00:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:00:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:01 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:16 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:25 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:30 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:40 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:01:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:01:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:01:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:01:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:01:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:01:55 mds1 multipathd: 8:80: mark as failed
Jan 16 09:01:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:55 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:01:55 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:01:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:01:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:01:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:01:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:01:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:01:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:01:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:01:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:00 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:15 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:26 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:30 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:41 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:45 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:02:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:02:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:02:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:02:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:02:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:02:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:02:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:02:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:02:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:02:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:02:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:02:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:02:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:02:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:02:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:01 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:16 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:26 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:30 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:41 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:51 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:03:55 mds1 init: tty (/dev/tty1) main process (3775) killed by TERM signal
Jan 16 09:03:55 mds1 init: tty (/dev/tty2) main process (3777) killed by TERM signal
Jan 16 09:03:55 mds1 init: tty (/dev/tty3) main process (3779) killed by TERM signal
Jan 16 09:03:55 mds1 init: tty (/dev/tty4) main process (3781) killed by TERM signal
Jan 16 09:03:55 mds1 init: tty (/dev/tty5) main process (3783) killed by TERM signal
Jan 16 09:03:55 mds1 init: tty (/dev/tty6) main process (3785) killed by TERM signal
Jan 16 09:03:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:03:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:03:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:03:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:03:55 mds1 abrtd: Got signal 15, exiting
Jan 16 09:03:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:03:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:03:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:03:59 mds1 acpid: exiting
Jan 16 09:03:59 mds1 multipathd: mdt2: event checker exit
Jan 16 09:03:59 mds1 multipathd: dm-2: remove map (uevent)
Jan 16 09:03:59 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 16 09:03:59 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 16 09:03:59 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 16 09:03:59 mds1 multipathd: dm-2: remove map (uevent)
Jan 16 09:03:59 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 16 09:03:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:03:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:03:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:03:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:03:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:03:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:03:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:03:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:00 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 16 09:04:00 mds1 kernel: Lustre: server umount MGS complete
Jan 16 09:04:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:01 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:16 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:18 mds1 kernel: Lustre: 3316:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389881051/real 1389881051]  req@ffff8808151eb400 x1457347450844692/t0(0) o400->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389881058 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 09:04:18 mds1 kernel: LustreError: 166-1: MGC192.168.1.40@o2ib: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Jan 16 09:04:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:21 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:24 mds1 kernel: Lustre: 3311:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389881058/real 1389881058]  req@ffff880812c5d000 x1457347450844796/t0(0) o250->MGC192.168.1.40@o2ib@0@lo:26/25 lens 400/544 e 0 to 1 dl 1389881064 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 09:04:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:26 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:31 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:40 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:43 mds1 kernel: Lustre: Evicted from MGS (at 192.168.1.42@o2ib) after server handle changed from 0x34b17884aac31ac1 to 0x86d1edc7b75710ef
Jan 16 09:04:43 mds1 kernel: Lustre: MGC192.168.1.40@o2ib: Connection restored to MGS (at 192.168.1.42@o2ib)
Jan 16 09:04:43 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 16 09:04:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:45 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:04:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:04:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:04:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:04:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:04:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:04:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:04:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:04:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:04:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:04:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:04:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:04:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:04:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:04:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:04:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:04:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:00 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:05 mds1 kernel: LNet: Service thread pid 5487 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 16 09:05:05 mds1 kernel: Pid: 5487, comm: mdt00_004
Jan 16 09:05:05 mds1 kernel: 
Jan 16 09:05:05 mds1 kernel: Call Trace:
Jan 16 09:05:05 mds1 kernel: [<ffffffffa010a1ad>] do_get_write_access+0x29d/0x520 [jbd2]
Jan 16 09:05:05 mds1 kernel: [<ffffffff811b59d7>] ? __find_get_block+0x97/0xe0
Jan 16 09:05:05 mds1 kernel: [<ffffffff81096de0>] ? wake_bit_function+0x0/0x50
Jan 16 09:05:05 mds1 kernel: [<ffffffffa010a581>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0baa2f8>] __ldiskfs_journal_get_write_access+0x38/0x80 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0bb6eb3>] ldiskfs_reserve_inode_write+0x73/0xa0 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffff811e0e4d>] ? dquot_mark_dquot_dirty+0x1d/0x80
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0bb6f2c>] ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0be670d>] ? ldiskfs_init_security+0x2d/0x90 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0bb3570>] ldiskfs_new_inode+0xd40/0x11c0 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0bca43a>] ldiskfs_create_inode+0x2a/0x90 [ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0c96596>] osd_mkfile+0x166/0x2a0 [osd_ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0edf3d3>] ? osp_txn_key_init+0x53/0x1a0 [osp]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0c96755>] osd_mkreg+0x25/0x60 [osd_ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0c9f66d>] osd_object_ea_create+0x21d/0x700 [osd_ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0e2b1af>] lod_object_create+0x13f/0x260 [lod]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0e87710>] mdd_object_create_internal+0xa0/0x1c0 [mdd]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0e98108>] mdd_create+0xa38/0x1730 [mdd]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0c99787>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0e28560>] ? lod_index_lookup+0x0/0x30 [lod]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d67dc8>] mdo_create+0x18/0x50 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d72031>] mdt_reint_open+0x1351/0x20a0 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0432e16>] ? upcall_cache_get_entry+0x296/0x880 [libcfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0597600>] ? lu_ucred_global_init+0x0/0x30 [obdclass]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d5aeb1>] mdt_reint_rec+0x41/0xe0 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d42c93>] mdt_reint_internal+0x4c3/0x780 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d4321d>] mdt_intent_reint+0x1ed/0x520 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 16 09:05:05 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 16 09:05:05 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 09:05:05 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 09:05:05 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 09:05:05 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 09:05:05 mds1 kernel: 
Jan 16 09:05:05 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389881105.5487
Jan 16 09:05:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:10 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:15 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:25 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:30 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:33 mds1 kernel: LNet: Service thread pid 4072 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 16 09:05:33 mds1 kernel: Pid: 4072, comm: mdt03_000
Jan 16 09:05:33 mds1 kernel: 
Jan 16 09:05:33 mds1 kernel: Call Trace:
Jan 16 09:05:33 mds1 kernel: [<ffffffff8150f362>] schedule_timeout+0x192/0x2e0
Jan 16 09:05:33 mds1 kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06cc8a0>] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06d10b1>] ldlm_completion_ast+0x4b1/0x920 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffff81063410>] ? default_wake_function+0x0/0x20
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06d0810>] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d40ccb>] mdt_object_lock0+0x33b/0xaf0 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d3b800>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06d0c00>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d41544>] mdt_object_lock+0x14/0x20 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d4f359>] mdt_getattr_name_lock+0xe19/0x1990 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06f8f85>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa071ff06>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06fb214>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d50169>] mdt_intent_getattr+0x299/0x480 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d3e8ce>] mdt_intent_policy+0x3ae/0x770 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06b1461>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa06da17f>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d3ed96>] mdt_enqueue+0x46/0xe0 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d45a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0d7fc55>] mds_regular_handle+0x15/0x20 [mdt]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa0709e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa04174ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa042827f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jan 16 09:05:33 mds1 kernel: [<ffffffffa07014c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Jan 16 09:05:33 mds1 kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Jan 16 09:05:33 mds1 kernel: [<ffffffffa070b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Jan 16 09:05:34 mds1 kernel: [<ffffffffa070a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Jan 16 09:05:34 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 09:05:34 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 09:05:34 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 09:05:34 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 09:05:34 mds1 kernel: 
Jan 16 09:05:34 mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1389881133.4072
Jan 16 09:05:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:40 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:45 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:45 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:45 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:05:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:05:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:05:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:05:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:05:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:05:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:05:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:05:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:05:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:05:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:05:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:05:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:05:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:05:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:05:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:05:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:01 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:01 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:04 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 192.168.1.55@o2ib (stopping)
Jan 16 09:06:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:10 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:10 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:10 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:16 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:25 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:30 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:30 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:30 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:35 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:40 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:40 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:40 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:40 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:40 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:40 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:44 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:44 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:44 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:44 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:44 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:44 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:44 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:44 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:44 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:44 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:45 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:45 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:45 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:45 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:45 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:45 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:45 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:45 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:49 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:49 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:49 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:49 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:49 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:49 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:49 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:49 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:49 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:49 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:50 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:50 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:50 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:50 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:50 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:50 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:50 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:50 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:50 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:50 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:50 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:54 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 192.168.1.55@o2ib (stopping)
Jan 16 09:06:54 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:54 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:54 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:54 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:54 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:54 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:54 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:54 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:54 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:54 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:06:55 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:55 mds1 multipathd: 8:80: reinstated
Jan 16 09:06:55 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:06:55 mds1 multipathd: 8:32: mark as failed
Jan 16 09:06:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:55 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:06:55 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:55 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:55 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:06:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:06:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:06:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:06:59 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:06:59 mds1 multipathd: 8:32: reinstated
Jan 16 09:06:59 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:06:59 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:06:59 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:59 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:06:59 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:06:59 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:59 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:06:59 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:00 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:00 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:00 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:00 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:00 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:00 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:00 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:00 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:00 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:00 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:00 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:01 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:04 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:04 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:04 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:04 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:04 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:04 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:04 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:04 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:04 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:04 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:05 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:05 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:05 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:05 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:05 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:05 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:05 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:05 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:09 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:09 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:09 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:09 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:09 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:09 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:09 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:09 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:09 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:09 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:10 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:10 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:10 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:10 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:10 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:10 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:10 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:10 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:14 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:14 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:14 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:14 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:14 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:14 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:14 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:14 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:14 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:14 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:15 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:15 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:15 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:15 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:15 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:15 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:15 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:15 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:15 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:15 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:15 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:19 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:19 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:19 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:19 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:19 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:19 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:19 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:19 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:19 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:19 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:20 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:20 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:20 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:20 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:20 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:20 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:20 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:20 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:20 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:20 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:20 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:24 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:24 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:24 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:24 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:24 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:24 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:24 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:24 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:24 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:24 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:25 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:25 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:25 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:25 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:25 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:25 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:25 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:25 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:25 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:25 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:25 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:29 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:29 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:29 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:29 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:29 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:29 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:29 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:29 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:29 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:29 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:30 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:30 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:30 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:30 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:30 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:30 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:30 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:30 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:31 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:31 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:31 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:34 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:34 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:34 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:34 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:34 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:34 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:34 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:34 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:34 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:34 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:35 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:35 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:35 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:35 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:35 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:35 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:35 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:35 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:35 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:35 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:35 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:39 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:39 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:39 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:39 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:39 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:39 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:39 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:39 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:39 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:39 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:40 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:40 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:40 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:40 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:40 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:41 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:44 mds1 kernel: Lustre: emc-MDT0000: Not available for connect from 192.168.1.55@o2ib (stopping)
Jan 16 09:07:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:45 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:46 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:46 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:46 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:46 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:46 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:46 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:46 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:50 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:50 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:50 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:50 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:50 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:50 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:50 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:50 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:50 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:50 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:51 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:51 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:51 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:51 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:51 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:51 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:51 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:51 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:51 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:51 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:51 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:51 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:51 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:55 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:55 mds1 multipathd: 8:32: reinstated
Jan 16 09:07:55 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:07:55 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:07:55 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:55 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:07:55 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:55 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:55 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:55 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:07:56 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:07:56 mds1 multipathd: 8:80: reinstated
Jan 16 09:07:56 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:07:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:56 mds1 multipathd: 8:32: mark as failed
Jan 16 09:07:56 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:07:56 mds1 multipathd: 8:80: mark as failed
Jan 16 09:07:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:07:56 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:07:56 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:07:56 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:07:56 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:56 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:07:56 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:07:56 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:00 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:00 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:00 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:00 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:00 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:00 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:00 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:00 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:00 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:00 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:01 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:01 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:01 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:01 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:01 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:01 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:01 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:01 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:01 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:01 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:02 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:02 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:02 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:05 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:05 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:05 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:05 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:05 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:05 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:05 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:05 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:05 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:05 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:06 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:06 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:06 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:06 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:06 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:06 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:06 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:06 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:06 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:06 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:06 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:06 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:06 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:10 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:10 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:10 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:10 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:10 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:10 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:10 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:10 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:10 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:10 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:11 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:11 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:11 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:11 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:11 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:11 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:11 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:11 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:11 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:11 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:11 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:11 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:11 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:15 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:15 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:15 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:15 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:15 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:15 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:15 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:15 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:15 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:15 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:16 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:16 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:16 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:16 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:16 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:16 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:16 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:16 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:16 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:16 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:16 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:16 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:16 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:20 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:20 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:20 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:20 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:20 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:20 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:20 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:20 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:20 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:20 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:21 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:21 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:21 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:21 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:21 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:21 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:21 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:21 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:21 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:21 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:21 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:21 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:21 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:25 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:25 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:25 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:25 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:25 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:25 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:25 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:25 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:25 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:25 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:26 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:26 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:26 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:26 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:26 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:26 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:26 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:26 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:26 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:26 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:26 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:26 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:26 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:30 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:30 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:30 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:30 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:30 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:30 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:30 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:30 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:30 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:30 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:31 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:31 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:31 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:31 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:31 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:31 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:31 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:31 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:31 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:31 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:32 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:32 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:32 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:35 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:35 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:35 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:35 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:35 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:35 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:35 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:35 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:35 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:35 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:36 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:36 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:36 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:36 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:36 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:36 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:36 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:36 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:36 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:36 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:36 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:36 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:36 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:40 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:40 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:40 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:40 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:40 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:40 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:40 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:40 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:40 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:40 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:41 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:41 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:41 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:41 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:41 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:41 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:41 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:41 mds1 multipathd: mdt1: remaining active paths: 0
Jan 16 09:08:41 mds1 kernel: sd 2:0:0:1: emc: long trespass command sent
Jan 16 09:08:41 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:41 mds1 kernel: sd 2:0:0:1: emc: at SP B Port 8 (owned, default SP A)
Jan 16 09:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:41 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:41 mds1 kernel: device-mapper: multipath: Failing path 8:80.
Jan 16 09:08:42 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:45 mds1 multipathd: mdt1: sdc - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:45 mds1 multipathd: 8:32: reinstated
Jan 16 09:08:45 mds1 multipathd: mdt1: queue_if_no_path enabled
Jan 16 09:08:45 mds1 multipathd: mdt1: Recovered to normal mode
Jan 16 09:08:45 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:45 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:08:45 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:45 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:45 mds1 kernel: blk_rq_check_limits: over max size limit.
Jan 16 09:08:45 mds1 kernel: device-mapper: multipath: Failing path 8:32.
Jan 16 09:08:46 mds1 multipathd: mdt1: sdf - emc_clariion_checker: Passive path is healthy.
Jan 16 09:08:46 mds1 multipathd: 8:80: reinstated
Jan 16 09:08:46 mds1 multipathd: mdt1: remaining active paths: 2
Jan 16 09:08:46 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:08:46 mds1 multipathd: 8:32: mark as failed
Jan 16 09:08:46 mds1 multipathd: mdt1: remaining active paths: 1
Jan 16 09:08:46 mds1 multipathd: 8:80: mark as failed
Jan 16 09:08:46 mds1 multipathd: mdt1: Entering recovery mode: max_retries=60
Jan 16 09:08:46 mds1 multipathd: mdt1: remaining acJan 16 09:12:02 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 16 09:12:02 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3137" x-info="http://www.rsyslog.com"] start
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys cpuset
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys cpu
Jan 16 09:12:02 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 16 09:12:02 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 16 09:12:02 mds1 kernel: KERNEL supported cpus:
Jan 16 09:12:02 mds1 kernel:  Intel GenuineIntel
Jan 16 09:12:02 mds1 kernel:  AMD AuthenticAMD
Jan 16 09:12:02 mds1 kernel:  Centaur CentaurHauls
Jan 16 09:12:02 mds1 kernel: BIOS-provided physical RAM map:
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 16 09:12:02 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 16 09:12:02 mds1 kernel: DMI 2.6 present.
Jan 16 09:12:02 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 16 09:12:02 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 16 09:12:02 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 16 09:12:02 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 16 09:12:02 mds1 kernel: Using GB pages for direct mapping
Jan 16 09:12:02 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 16 09:12:02 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 16 09:12:02 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 16 09:12:02 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 16 09:12:02 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 16 09:12:02 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 16 09:12:02 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 16 09:12:02 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 16 09:12:02 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 16 09:12:02 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 16 09:12:02 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 16 09:12:02 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 16 09:12:02 mds1 kernel: Setting APIC routing to flat.
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 16 09:12:02 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 16 09:12:02 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 16 09:12:02 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 16 09:12:02 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 16 09:12:02 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 16 09:12:02 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 16 09:12:02 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 16 09:12:02 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 16 09:12:02 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 16 09:12:02 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 16 09:12:02 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 16 09:12:02 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 16 09:12:02 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 16 09:12:02 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 16 09:12:02 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 16 09:12:02 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 16 09:12:02 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 16 09:12:02 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 16 09:12:02 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 16 09:12:02 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 16 09:12:02 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 16 09:12:02 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 16 09:12:02 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 16 09:12:02 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 16 09:12:02 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 16 09:12:02 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 16 09:12:02 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 16 09:12:02 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 16 09:12:02 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 16 09:12:02 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 16 09:12:02 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 16 09:12:02 mds1 kernel: Zone PFN ranges:
Jan 16 09:12:02 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 16 09:12:02 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 16 09:12:02 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 16 09:12:02 mds1 kernel: Movable zone start PFN for each node
Jan 16 09:12:02 mds1 kernel: early_node_map[5] active PFN ranges
Jan 16 09:12:02 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 16 09:12:02 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 16 09:12:02 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 16 09:12:02 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 16 09:12:02 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 16 09:12:02 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 16 09:12:02 mds1 kernel: Setting APIC routing to flat.
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 16 09:12:02 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 16 09:12:02 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 16 09:12:02 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 16 09:12:02 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 16 09:12:02 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 16 09:12:02 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 16 09:12:02 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 16 09:12:02 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 16 09:12:02 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 16 09:12:02 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 16 09:12:02 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 16 09:12:02 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 16 09:12:02 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 16 09:12:02 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 16 09:12:02 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 16 09:12:02 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 16 09:12:02 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 16 09:12:02 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 16 09:12:02 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 16 09:12:02 mds1 kernel: Policy zone: Normal
Jan 16 09:12:02 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 16 09:12:02 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 16 09:12:02 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 16 09:12:02 mds1 kernel: Checking aperture...
Jan 16 09:12:02 mds1 kernel: No AGP bridge found
Jan 16 09:12:02 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 16 09:12:02 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 16 09:12:02 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 16 09:12:02 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 16 09:12:02 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 16 09:12:02 mds1 kernel: Hierarchical RCU implementation.
Jan 16 09:12:02 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 16 09:12:02 mds1 kernel: Extended CMOS year: 2000
Jan 16 09:12:02 mds1 kernel: Console: colour VGA+ 80x25
Jan 16 09:12:02 mds1 kernel: console [tty0] enabled
Jan 16 09:12:02 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 16 09:12:02 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 16 09:12:02 mds1 kernel: Fast TSC calibration using PIT
Jan 16 09:12:02 mds1 kernel: Detected 1995.132 MHz processor.
Jan 16 09:12:02 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.26 BogoMIPS (lpj=1995132)
Jan 16 09:12:02 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 16 09:12:02 mds1 kernel: Security Framework initialized
Jan 16 09:12:02 mds1 kernel: SELinux:  Initializing.
Jan 16 09:12:02 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 16 09:12:02 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 16 09:12:02 mds1 kernel: Mount-cache hash table entries: 256
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys ns
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys memory
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys devices
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys freezer
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys net_cls
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys blkio
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys perf_event
Jan 16 09:12:02 mds1 kernel: Initializing cgroup subsys net_prio
Jan 16 09:12:02 mds1 kernel: CPU: Physical Processor ID: 0
Jan 16 09:12:02 mds1 kernel: CPU: Processor Core ID: 0
Jan 16 09:12:02 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 16 09:12:02 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 16 09:12:02 mds1 kernel: using mwait in idle threads.
Jan 16 09:12:02 mds1 kernel: ACPI: Core revision 20090903
Jan 16 09:12:02 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 16 09:12:02 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 16 09:12:02 mds1 kernel: dmar: Host address width 46
Jan 16 09:12:02 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 16 09:12:02 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 16 09:12:02 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 16 09:12:02 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 16 09:12:02 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 16 09:12:02 mds1 kernel: dmar: ATSR flags: 0x0
Jan 16 09:12:02 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 16 09:12:02 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 16 09:12:02 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 16 09:12:02 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 16 09:12:02 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 16 09:12:02 mds1 kernel: Enabling x2apic
Jan 16 09:12:02 mds1 kernel: Enabled x2apic
Jan 16 09:12:02 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 16 09:12:02 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 16 09:12:02 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 16 09:12:02 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 16 09:12:02 mds1 kernel: PEBS disabled due to CPU errata.
Jan 16 09:12:02 mds1 kernel: ... version:                3
Jan 16 09:12:02 mds1 kernel: ... bit width:              48
Jan 16 09:12:02 mds1 kernel: ... generic registers:      8
Jan 16 09:12:02 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 16 09:12:02 mds1 kernel: ... max period:             000000007fffffff
Jan 16 09:12:02 mds1 kernel: ... fixed-purpose events:   3
Jan 16 09:12:02 mds1 kernel: ... event mask:             00000007000000ff
Jan 16 09:12:02 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 16 09:12:02 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 16 09:12:02 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 16 09:12:02 mds1 kernel: Brought up 12 CPUs
Jan 16 09:12:02 mds1 kernel: Total of 12 processors activated (47880.66 BogoMIPS).
Jan 16 09:12:02 mds1 kernel: devtmpfs: initialized
Jan 16 09:12:02 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 16 09:12:02 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 16 09:12:02 mds1 kernel: regulator: core version 0.5
Jan 16 09:12:02 mds1 kernel: NET: Registered protocol family 16
Jan 16 09:12:02 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 16 09:12:02 mds1 kernel: ACPI: bus type pci registered
Jan 16 09:12:02 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 16 09:12:02 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 16 09:12:02 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 16 09:12:02 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 16 09:12:02 mds1 kernel: bio: create slab <bio-0> at 0
Jan 16 09:12:02 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 16 09:12:02 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 16 09:12:02 mds1 kernel: ACPI: Interpreter enabled
Jan 16 09:12:02 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 16 09:12:02 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 16 09:12:02 mds1 kernel: ACPI: No dock devices found.
Jan 16 09:12:02 mds1 kernel: HEST: Table parsing has been initialized.
Jan 16 09:12:02 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 16 09:12:02 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 16 09:12:02 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 16 09:12:02 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 16 09:12:02 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 16 09:12:02 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 16 09:12:02 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 16 09:12:02 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 16 09:12:02 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 16 09:12:02 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 16 09:12:02 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 16 09:12:02 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 16 09:12:02 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 16 09:12:02 mds1 kernel: vgaarb: loaded
Jan 16 09:12:02 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 16 09:12:02 mds1 kernel: SCSI subsystem initialized
Jan 16 09:12:02 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 16 09:12:02 mds1 kernel: usbcore: registered new interface driver hub
Jan 16 09:12:02 mds1 kernel: usbcore: registered new device driver usb
Jan 16 09:12:02 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 16 09:12:02 mds1 kernel: NetLabel: Initializing
Jan 16 09:12:02 mds1 kernel: NetLabel:  domain hash size = 128
Jan 16 09:12:02 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 16 09:12:02 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 16 09:12:02 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 16 09:12:02 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 16 09:12:02 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 16 09:12:02 mds1 kernel: Switching to clocksource hpet
Jan 16 09:12:02 mds1 kernel: pnp: PnP ACPI init
Jan 16 09:12:02 mds1 kernel: ACPI: bus type pnp registered
Jan 16 09:12:02 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 16 09:12:02 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 16 09:12:02 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 16 09:12:02 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 16 09:12:02 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 16 09:12:02 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 16 09:12:02 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 16 09:12:02 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 16 09:12:02 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 16 09:12:02 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 16 09:12:02 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 16 09:12:02 mds1 kernel: NET: Registered protocol family 2
Jan 16 09:12:02 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 16 09:12:02 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 16 09:12:02 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 16 09:12:02 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 16 09:12:02 mds1 kernel: TCP reno registered
Jan 16 09:12:02 mds1 kernel: NET: Registered protocol family 1
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 16 09:12:02 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 16 09:12:02 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 16 09:12:02 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 16 09:12:02 mds1 kernel: dmar: Unsupported device scope
Jan 16 09:12:02 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 16 09:12:02 mds1 kernel: type=2000 audit(1389881500.055:1): initialized
Jan 16 09:12:02 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 16 09:12:02 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 16 09:12:02 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 16 09:12:02 mds1 kernel: msgmni has been set to 32768
Jan 16 09:12:02 mds1 kernel: alg: No test for stdrng (krng)
Jan 16 09:12:02 mds1 kernel: ksign: Installing public key data
Jan 16 09:12:02 mds1 kernel: Loading keyring
Jan 16 09:12:02 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 16 09:12:02 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 16 09:12:02 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 16 09:12:02 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 16 09:12:02 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 16 09:12:02 mds1 kernel: io scheduler noop registered
Jan 16 09:12:02 mds1 kernel: io scheduler anticipatory registered
Jan 16 09:12:02 mds1 kernel: io scheduler deadline registered (default)
Jan 16 09:12:02 mds1 kernel: io scheduler cfq registered
Jan 16 09:12:02 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 16 09:12:02 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 16 09:12:02 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 16 09:12:02 mds1 kernel: acpiphp: Slot [2] registered
Jan 16 09:12:02 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 16 09:12:02 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 16 09:12:02 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 16 09:12:02 mds1 kernel: ipmi message handler version 39.2
Jan 16 09:12:02 mds1 kernel: IPMI System Interface driver.
Jan 16 09:12:02 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 16 09:12:02 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 16 09:12:02 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 16 09:12:02 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 16 09:12:02 mds1 kernel: IPMI kcs interface initialized
Jan 16 09:12:02 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 16 09:12:02 mds1 kernel: ACPI: Power Button [PWRF]
Jan 16 09:12:02 mds1 kernel: ERST: Could not register with persistent store
Jan 16 09:12:02 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 16 09:12:02 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 16 09:12:02 mds1 kernel: Non-volatile memory driver v1.3
Jan 16 09:12:02 mds1 kernel: Linux agpgart interface v0.103
Jan 16 09:12:02 mds1 kernel: crash memory driver: version 1.1
Jan 16 09:12:02 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 16 09:12:02 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 16 09:12:02 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 16 09:12:02 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 16 09:12:02 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 16 09:12:02 mds1 kernel: brd: module loaded
Jan 16 09:12:02 mds1 kernel: loop: module loaded
Jan 16 09:12:02 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 16 09:12:02 mds1 kernel: Fixed MDIO Bus: probed
Jan 16 09:12:02 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 16 09:12:02 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 16 09:12:02 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 16 09:12:02 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 16 09:12:02 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 16 09:12:02 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 16 09:12:02 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 16 09:12:02 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 16 09:12:02 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 16 09:12:02 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 16 09:12:02 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 16 09:12:02 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 16 09:12:02 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 16 09:12:02 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 16 09:12:02 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 16 09:12:02 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 16 09:12:02 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 16 09:12:02 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 16 09:12:02 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 16 09:12:02 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 16 09:12:02 mds1 kernel: Switching to clocksource tsc
Jan 16 09:12:02 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 16 09:12:02 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 16 09:12:02 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 16 09:12:02 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 16 09:12:02 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 16 09:12:02 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 16 09:12:02 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 16 09:12:02 mds1 kernel: cpuidle: using governor ladder
Jan 16 09:12:02 mds1 kernel: cpuidle: using governor menu
Jan 16 09:12:02 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 16 09:12:02 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 16 09:12:02 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 16 09:12:02 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 16 09:12:02 mds1 kernel: TCP cubic registered
Jan 16 09:12:02 mds1 kernel: Initializing XFRM netlink socket
Jan 16 09:12:02 mds1 kernel: NET: Registered protocol family 17
Jan 16 09:12:02 mds1 kernel: registered taskstats version 1
Jan 16 09:12:02 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-16 14:11:42 UTC (1389881502)
Jan 16 09:12:02 mds1 kernel: Initalizing network drop monitor service
Jan 16 09:12:02 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 16 09:12:02 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 16 09:12:02 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 16 09:12:02 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 16 09:12:02 mds1 kernel: dracut: dracut-004-303.el6
Jan 16 09:12:02 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 16 09:12:02 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 16 09:12:02 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 16 09:12:02 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 16 09:12:02 mds1 kernel: udev: starting version 147
Jan 16 09:12:02 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 16 09:12:02 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 16 09:12:02 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 16 09:12:02 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 16 09:12:02 mds1 kernel: dracut: Starting plymouth daemon
Jan 16 09:12:02 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 16 09:12:02 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 16 09:12:02 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 16 09:12:02 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 16 09:12:02 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 16 09:12:02 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 16 09:12:02 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 16 09:12:02 mds1 kernel: megasas: FW now in Ready state
Jan 16 09:12:02 mds1 kernel: megasas:IOC Init cmd success
Jan 16 09:12:02 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 16 09:12:02 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 16 09:12:02 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 16 09:12:02 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 16 09:12:02 mds1 kernel: megasas: INIT adapter done
Jan 16 09:12:02 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 16 09:12:02 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 16 09:12:02 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 16 09:12:02 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 16 09:12:02 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 16 09:12:02 mds1 kernel: scsi1 : qla2xxx
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 16 09:12:02 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 16 09:12:02 mds1 kernel: scsi2 : qla2xxx
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 16 09:12:02 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 16 09:12:02 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 16 09:12:02 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 16 09:12:02 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 16 09:12:02 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 16 09:12:02 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 16 09:12:02 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 16 09:12:02 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 16 09:12:02 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2f7f, mb[3]=6100, cmd=6a ****.
Jan 16 09:12:02 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2f4b, mb[3]=1100, cmd=6a ****.
Jan 16 09:12:02 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 16 09:12:02 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 16 09:12:02 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 16 09:12:02 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 16 09:12:02 mds1 kernel: scsi3 : ahci
Jan 16 09:12:02 mds1 kernel: scsi4 : ahci
Jan 16 09:12:02 mds1 kernel: scsi5 : ahci
Jan 16 09:12:02 mds1 kernel: scsi6 : ahci
Jan 16 09:12:02 mds1 kernel: scsi7 : ahci
Jan 16 09:12:02 mds1 kernel: scsi8 : ahci
Jan 16 09:12:02 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 16 09:12:02 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 16 09:12:02 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 16 09:12:02 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 16 09:12:02 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 16 09:12:02 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 16 09:12:02 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 16 09:12:02 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 16 09:12:02 mds1 kernel: ata1.00: configured for UDMA/100
Jan 16 09:12:02 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 16 09:12:02 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 16 09:12:02 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sda:
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 16 09:12:02 mds1 kernel: sdb: unknown partition table
Jan 16 09:12:02 mds1 kernel: sdc:
Jan 16 09:12:02 mds1 kernel: sdd:
Jan 16 09:12:02 mds1 kernel: sdf:
Jan 16 09:12:02 mds1 kernel: sde:
Jan 16 09:12:02 mds1 kernel: sdg: unknown partition table
Jan 16 09:12:02 mds1 kernel: unknown partition table
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: unknown partition table
Jan 16 09:12:02 mds1 kernel: unknown partition table
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: unknown partition table
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: sda1 sda2 sda3
Jan 16 09:12:02 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 16 09:12:02 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 16 09:12:02 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 16 09:12:02 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 16 09:12:02 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 16 09:12:02 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 16 09:12:02 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 16 09:12:02 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 16 09:12:02 mds1 kernel: SELinux:  Disabled at runtime.
Jan 16 09:12:02 mds1 kernel: type=1404 audit(1389881505.507:2): selinux=0 auid=4294967295 ses=4294967295
Jan 16 09:12:02 mds1 kernel: dracut: 
Jan 16 09:12:02 mds1 kernel: dracut: Switching root
Jan 16 09:12:02 mds1 kernel: udev: starting version 147
Jan 16 09:12:02 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 16 09:12:02 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 16 09:12:02 mds1 kernel: PTP clock support registered
Jan 16 09:12:02 mds1 kernel: dca service started, version 1.12.1
Jan 16 09:12:02 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 16 09:12:02 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 16 09:12:02 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 16 09:12:02 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 16 09:12:02 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 16 09:12:02 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 16 09:12:02 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 16 09:12:02 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 16 09:12:02 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 16 09:12:02 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 16 09:12:02 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 16 09:12:02 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 16 09:12:02 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 16 09:12:02 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 16 09:12:02 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 16 09:12:02 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 16 09:12:02 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 16 09:12:02 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 16 09:12:02 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 16 09:12:02 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 16 09:12:02 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 16 09:12:02 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 16 09:12:02 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 16 09:12:02 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 16 09:12:02 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 16 09:12:02 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 16 09:12:02 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 16 09:12:02 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 16 09:12:02 mds1 kernel: emc: device handler registered
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (bound, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: long trespass command sent
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 16 09:12:02 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 16 09:12:02 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 16 09:12:02 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 16 09:12:02 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 16 09:12:02 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 16 09:12:02 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 16 09:12:02 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 16 09:12:02 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028628000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 16 09:12:02 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x1028628000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 16 09:12:02 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 16 09:12:02 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 16 09:12:02 mds1 kernel: NET: Registered protocol family 10
Jan 16 09:12:02 mds1 kernel: lo: Disabled Privacy Extensions
Jan 16 09:12:02 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 16 09:12:02 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 16 09:12:02 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 16 09:12:02 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 16 09:12:02 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 16 09:12:02 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 16 09:12:02 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 16 09:12:02 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 16 09:12:02 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 16 09:12:02 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 16 09:12:02 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 16 09:12:02 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 16 09:12:02 mds1 rpc.statd[3232]: Version 1.2.3 starting
Jan 16 09:12:02 mds1 sm-notify[3233]: Version 1.2.3 starting
Jan 16 09:12:03 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 16 09:12:03 mds1 kernel: RPC: Registered udp transport module.
Jan 16 09:12:03 mds1 kernel: RPC: Registered tcp transport module.
Jan 16 09:12:03 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 16 09:12:03 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 16 09:12:03 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 16 09:12:03 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 16 09:12:03 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 16 09:12:07 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 16 09:12:07 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 16 09:12:11 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 16 09:12:11 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 16 09:12:12 mds1 cib[3375]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 attrd[3378]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 lrmd[3377]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 stonith-ng[3376]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 pengine[3379]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 crmd[3380]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 16 09:12:12 mds1 logger[3389]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 16 09:12:14 mds1 kdump: kexec: loaded kdump kernel
Jan 16 09:12:14 mds1 kdump: started up
Jan 16 09:12:14 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 16 09:12:14 mds1 acpid: starting up
Jan 16 09:12:14 mds1 acpid: 1 rule loaded
Jan 16 09:12:14 mds1 acpid: waiting for events: event logging is off
Jan 16 09:12:15 mds1 acpid: client connected from 3593[68:68]
Jan 16 09:12:15 mds1 acpid: 1 client rule loaded
Jan 16 09:12:15 mds1 kernel: ipmi device interface
Jan 16 09:12:16 mds1 automount[3632]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 16 09:12:17 mds1 abrtd: Init complete, entering main loop
Jan 16 09:12:17 mds1 cimserver[3783]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 16 09:12:23 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 16 09:12:25 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 16 09:12:55 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 16 09:12:55 mds1 kernel: 
Jan 16 09:12:55 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 16 09:12:55 mds1 kernel: 
Jan 16 09:12:56 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 16 09:12:56 mds1 kernel: 
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 16 09:13:38 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 16 09:13:44 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389881619/real 0]  req@ffff880fdb377c00 x1457396416249864/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389881624 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 09:13:44 mds1 kernel: LustreError: 4146:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -5 (not fatal)
Jan 16 09:13:44 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 16 09:13:44 mds1 kernel: LustreError: 4386:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 16 09:13:44 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 16 09:13:44 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 16 09:13:44 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 16 09:13:44 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 16 09:13:44 mds1 kernel: LustreError: 4454:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 16 09:13:44 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 16 09:13:49 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389881624/real 0]  req@ffff880fdb377000 x1457396416249884/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389881629 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 09:14:09 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 4 clients reconnect
Jan 16 09:14:15 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389881649/real 0]  req@ffff88100f3c2800 x1457396416250704/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389881655 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 09:14:34 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 4 clients reconnect
Jan 16 09:15:33 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client 82e429b3-24fe-4012-c167-024517159089 (at 192.168.1.55@o2ib), waiting for all 4 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 4:00
Jan 16 09:15:58 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client 82e429b3-24fe-4012-c167-024517159089 (at 192.168.1.55@o2ib), waiting for all 4 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 3:35
Jan 16 09:15:58 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 16 09:16:01 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 16 09:16:01 mds1 kernel: LustreError: 4965:0:(ldlm_lib.c:2129:target_stop_recovery_thread()) emc-MDT0001: Aborting recovery
Jan 16 09:16:01 mds1 kernel: Lustre: 4506:0:(ldlm_lib.c:1798:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jan 16 09:16:01 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 1:52, of 4 clients 0 recovered and 0 were evicted.
Jan 16 09:16:01 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID (at 0@lo) reconnecting, waiting for 4 clients in recovery for 3:33
Jan 16 09:16:01 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID (at 0@lo) refused reconnection, still busy with 1 active RPCs
Jan 16 09:16:01 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -16.
Jan 16 09:16:01 mds1 kernel: LustreError: 4470:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0000-osc-MDT0001: can't precreate: rc = -5
Jan 16 09:16:01 mds1 kernel: LustreError: 4470:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0000-osc-MDT0001: cannot precreate objects: rc = -5
Jan 16 09:16:01 mds1 kernel: LustreError: 4476:0:(osp_precreate.c:481:osp_precreate_send()) emc-OST0003-osc-MDT0001: can't precreate: rc = -5
Jan 16 09:16:01 mds1 kernel: LustreError: 4476:0:(osp_precreate.c:481:osp_precreate_send()) Skipped 2 previous similar messages
Jan 16 09:16:01 mds1 kernel: LustreError: 4476:0:(osp_precreate.c:986:osp_precreate_thread()) emc-OST0003-osc-MDT0001: cannot precreate objects: rc = -5
Jan 16 09:16:01 mds1 kernel: LustreError: 4476:0:(osp_precreate.c:986:osp_precreate_thread()) Skipped 2 previous similar messages
Jan 16 09:16:01 mds1 kernel: Lustre: server umount emc-MDT0001 complete
Jan 16 09:16:03 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:16:10 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.55@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 16 09:16:23 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client 82e429b3-24fe-4012-c167-024517159089 (at 192.168.1.55@o2ib), waiting for all 4 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 3:10
Jan 16 09:16:37 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389881649/real 1389881649]  req@ffff88100f147800 x1457396416250808/t0(0) o400->emc-MDT0001-osp-MDT0000@0@lo:24/10 lens 224/224 e 5 to 1 dl 1389881797 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 16 09:16:37 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 16 09:16:48 mds1 kernel: Lustre: emc-MDT0000: Denying connection for new client 82e429b3-24fe-4012-c167-024517159089 (at 192.168.1.55@o2ib), waiting for all 4 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 2:45
Jan 16 09:16:53 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:17:41 mds1 kernel: INFO: task tgt_recov:4453 blocked for more than 120 seconds.
Jan 16 09:17:41 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 09:17:41 mds1 kernel: tgt_recov     D 0000000000000009     0  4453      2 0x00000080
Jan 16 09:17:41 mds1 kernel: ffff880fd6669da0 0000000000000046 0000000000000000 ffff88085c456740
Jan 16 09:17:41 mds1 kernel: ffff880fd6669d30 ffffffff81055f96 ffff880fd6669d40 ffff88102e4a7540
Jan 16 09:17:41 mds1 kernel: ffff880fd6667ab8 ffff880fd6669fd8 000000000000fb88 ffff880fd6667ab8
Jan 16 09:17:41 mds1 kernel: Call Trace:
Jan 16 09:17:41 mds1 kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06ba210>] ? check_for_clients+0x0/0x70 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06bb87d>] target_recovery_overseer+0x9d/0x230 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06b9f00>] ? exp_connect_healthy+0x0/0x20 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06c2680>] target_recovery_thread+0x540/0x1920 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffff81063422>] ? default_wake_function+0x12/0x20
Jan 16 09:17:41 mds1 kernel: [<ffffffffa06c2140>] ? target_recovery_thread+0x0/0x1920 [ptlrpc]
Jan 16 09:17:41 mds1 kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Jan 16 09:17:41 mds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jan 16 09:17:41 mds1 kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan 16 09:17:41 mds1 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan 16 09:17:43 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:18:33 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:19:23 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:19:34 mds1 kernel: Lustre: emc-MDT0000: recovery is timed out, evict stale exports
Jan 16 09:19:34 mds1 kernel: Lustre: emc-MDT0000: disconnecting 3 stale clients
Jan 16 09:19:34 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 5:00, of 4 clients 1 recovered and 3 were evicted.
Jan 16 09:20:13 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0001-mdtlov_UUID seen on new nid 192.168.1.42@o2ib when existing nid 0@lo is already connected
Jan 16 09:20:54 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0001-mdtlov_UUID (at 0@lo) in 293 seconds. I think it's dead, and I am evicting it. exp ffff880fd738c800, cur 1389882054 expire 1389881904 last 1389881761
Jan 16 09:21:48 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 16 09:26:10 mds1 kernel: Lustre: Echo OBD driver; http://www.lustre.org/
Jan 16 09:26:10 mds1 kernel: LustreError: 5906:0:(echo_client.c:1743:echo_md_lookup()) lookup tests: rc = -2
Jan 16 09:26:10 mds1 kernel: LustreError: 5906:0:(echo_client.c:1942:echo_md_destroy_internal()) Can't find child tests: rc = -2
Jan 16 09:26:10 mds1 kernel: LustreError: 5908:0:(echo_client.c:1743:echo_md_lookup()) lookup tests1: rc = -2
Jan 16 09:26:10 mds1 kernel: LustreError: 5908:0:(echo_client.c:1942:echo_md_destroy_internal()) Can't find child tests1: rc = -2
Jan 16 12:55:08 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client ab938b99-db5f-8a79-e348-b0c047e4c896 (at 192.168.1.57@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8807e1aba400, cur 1389894908 expire 1389894758 last 1389894680
Jan 16 13:04:50 mds1 cibadmin[25851]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 13:10:44 mds1 cibadmin[26355]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 13:47:37 mds1 cibadmin[29544]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 14:03:11 mds1 cibadmin[30989]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 14:20:40 mds1 cpuspeed: Disabling performance cpu frequency scaling governor
Jan 16 14:20:40 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 16 14:33:12 mds1 cpuspeed: Disabling ondemand cpu frequency scaling governor
Jan 16 14:33:12 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 16 14:43:52 mds1 cibadmin[34815]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 14:51:31 mds1 cibadmin[35427]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 14:56:05 mds1 cibadmin[35896]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 14:56:09 mds1 cibadmin[35900]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 15:32:59 mds1 kernel: LustreError: 11-0: emc-OST0005-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 16 15:32:59 mds1 kernel: Lustre: emc-OST000a-osc-MDT0000: Connection to emc-OST000a (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:32:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 16 15:32:59 mds1 kernel: LustreError: Skipped 4 previous similar messages
Jan 16 15:32:59 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection to emc-OST0005 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:32:59 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 16 15:33:06 mds1 kernel: Lustre: 3356:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389904379/real 1389904379]  req@ffff880fd6413c00 x1457396424879708/t0(0) o13->emc-OST0007-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389904386 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:33:06 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection to emc-OST0007 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:33:06 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 16 15:33:07 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389904380/real 1389904380]  req@ffff88100fe51000 x1457396424879724/t0(0) o13->emc-OST0002-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389904387 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:33:07 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 16 15:33:07 mds1 kernel: Lustre: 3350:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389904380/real 1389904380]  req@ffff880feb5c2000 x1457396424879732/t0(0) o13->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389904387 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:33:07 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:33:07 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 16 15:33:12 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389904386/real 1389904386]  req@ffff880ff5e55800 x1457396424879844/t0(0) o8->emc-OST0003-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389904392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 15:33:14 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection restored to emc-OST0004 (at 192.168.1.46@o2ib)
Jan 16 15:33:14 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection restored to emc-OST0008 (at 192.168.1.46@o2ib)
Jan 16 15:33:37 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 16 15:33:37 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 16 15:45:37 mds1 kernel: LustreError: 11-0: emc-OST0002-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 15:45:37 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 16 15:45:37 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:45:37 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 15:45:39 mds1 kernel: LustreError: 11-0: emc-OST0004-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 15:45:39 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection to emc-OST0004 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:45:39 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 16 15:45:46 mds1 kernel: Lustre: MGS: Client e16eca6b-eeb5-6c47-5f4b-5b6abd110532 (at 192.168.1.44@o2ib) reconnecting
Jan 16 15:45:54 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection restored to emc-OST0004 (at 192.168.1.44@o2ib)
Jan 16 15:45:54 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 16 15:55:04 mds1 cibadmin[40980]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 15:55:41 mds1 kernel: Lustre: 3351:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389905734/real 1389905734]  req@ffff88081c940800 x1457396424906648/t0(0) o13->emc-OST000e-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389905741 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:55:41 mds1 kernel: Lustre: 3351:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jan 16 15:55:41 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:55:41 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 16 15:55:42 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389905735/real 1389905735]  req@ffff880ff9181800 x1457396424906672/t0(0) o13->emc-OST0011-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389905742 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:55:42 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jan 16 15:55:43 mds1 kernel: Lustre: 3355:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389905735/real 1389905735]  req@ffff88081da0ac00 x1457396424906664/t0(0) o13->emc-OST0015-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389905742 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 15:55:43 mds1 kernel: Lustre: 3355:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jan 16 15:55:43 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 15:55:43 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 16 15:55:47 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389905741/real 1389905741]  req@ffff88082549d000 x1457396424906776/t0(0) o8->emc-OST0010-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389905747 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 15:55:47 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection restored to emc-OST0010 (at 192.168.1.44@o2ib)
Jan 16 15:55:47 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 16 15:55:49 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389905743/real 1389905743]  req@ffff880830855400 x1457396424906820/t0(0) o8->emc-OST0015-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389905749 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 15:55:49 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Jan 16 15:56:12 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection restored to emc-OST0015 (at 192.168.1.44@o2ib)
Jan 16 15:56:12 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 16 16:03:37 mds1 kernel: Lustre: 3358:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389906209/real 0]  req@ffff880824588800 x1457396424918436/t0(0) o13->emc-OST0009-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389906217 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 16:03:37 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:03:47 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389906220/real 0]  req@ffff88082365f000 x1457396424918580/t0(0) o8->emc-OST0011-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389906227 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 16:03:47 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
Jan 16 16:04:12 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389906245/real 1389906245]  req@ffff880825147000 x1457396424918604/t0(0) o8->emc-OST0008-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389906252 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 16:04:12 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 16 16:04:39 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0014_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:04:40 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0004_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:04:40 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 16 16:04:42 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000d_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:04:42 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 16 16:04:49 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 16 16:04:49 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 16 16:05:12 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 16 16:05:12 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 16 16:05:33 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0014_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:05:33 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 16 16:05:58 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000d_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:05:58 mds1 kernel: Lustre: Skipped 17 previous similar messages
Jan 16 16:06:23 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0014_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 16 16:06:23 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 16 16:06:56 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0000_UUID (at 192.168.1.44@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808248ee000, cur 1389906416 expire 1389906266 last 1389906189
Jan 16 16:06:59 mds1 kernel: Lustre: MGS: haven't heard from client e16eca6b-eeb5-6c47-5f4b-5b6abd110532 (at 192.168.1.44@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff8808319a5000, cur 1389906419 expire 1389906269 last 1389906189
Jan 16 16:06:59 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 16 16:15:12 mds1 kernel: LustreError: 11-0: emc-OST0002-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 16:15:12 mds1 kernel: LustreError: Skipped 9 previous similar messages
Jan 16 16:15:12 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:15:12 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 16 16:15:14 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 16:15:14 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection to emc-OST0008 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:15:14 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 16 16:15:14 mds1 kernel: LustreError: Skipped 7 previous similar messages
Jan 16 16:15:25 mds1 kernel: Lustre: MGS: Client f4c65f19-4f49-de52-34a3-6e466abfbbd0 (at 192.168.1.44@o2ib) reconnecting
Jan 16 16:15:35 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection restored to emc-OST0009 (at 192.168.1.44@o2ib)
Jan 16 16:15:35 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 16 16:15:38 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 16 16:15:38 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 16 16:16:14 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389906931/real 1389906931]  req@ffff880ff49cbc00 x1457396424951652/t0(0) o400->emc-OST0012-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 224/224 e 0 to 1 dl 1389906974 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Jan 16 16:16:14 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 16 16:16:14 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection restored to emc-OST0010 (at 192.168.1.46@o2ib)
Jan 16 16:44:50 mds1 kernel: LustreError: 11-0: emc-OST000d-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 16:44:50 mds1 kernel: LustreError: Skipped 15 previous similar messages
Jan 16 16:44:50 mds1 kernel: Lustre: emc-OST000d-osc-MDT0000: Connection to emc-OST000d (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:44:50 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 16 16:44:50 mds1 kernel: LustreError: 11-0: emc-OST0012-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 16 16:44:50 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 16 16:44:50 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:44:50 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 16 16:44:58 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.44@o2ib)
Jan 16 16:44:58 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 16 16:45:02 mds1 kernel: Lustre: 3349:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389908694/real 1389908694]  req@ffff880ff5861400 x1457396425001960/t0(0) o13->emc-OST0010-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389908702 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 16:45:02 mds1 kernel: Lustre: 3349:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 16 16:45:02 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection to emc-OST0010 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:45:02 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection restored to emc-OST0014 (at 192.168.1.44@o2ib)
Jan 16 16:45:02 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 16 16:45:03 mds1 kernel: Lustre: 3354:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389908695/real 1389908695]  req@ffff880ff0c3a400 x1457396425001976/t0(0) o13->emc-OST0011-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389908703 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 16 16:45:05 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection restored to emc-OST0013 (at 192.168.1.44@o2ib)
Jan 16 16:45:08 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection restored to emc-OST0016 (at 192.168.1.44@o2ib)
Jan 16 16:45:08 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 16 16:45:09 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389908702/real 1389908702]  req@ffff88081d106800 x1457396425002148/t0(0) o8->emc-OST0010-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389908709 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 16:45:09 mds1 kernel: Lustre: 3348:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jan 16 16:45:37 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection restored to emc-OST0015 (at 192.168.1.44@o2ib)
Jan 16 16:45:37 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 16 16:48:26 mds1 cibadmin[45617]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 16 16:51:40 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 16 16:51:40 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 16 16:51:40 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:51:40 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 16 16:51:43 mds1 kernel: LustreError: 11-0: emc-OST0016-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 16 16:51:43 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection to emc-OST0016 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:51:45 mds1 kernel: LustreError: 11-0: emc-OST0013-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 16 16:51:45 mds1 kernel: LustreError: Skipped 6 previous similar messages
Jan 16 16:51:47 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 16 16:51:47 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 16 16:51:52 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 16 16:52:00 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection restored to emc-OST0015 (at 192.168.1.46@o2ib)
Jan 16 16:52:00 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 16 17:10:25 mds1 kernel: Lustre: 4649:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389910218/real 1389910218]  req@ffff88081b012000 x1457396425034304/t0(0) o104->emc-MDT0000@192.168.1.55@o2ib:15/16 lens 296/224 e 0 to 1 dl 1389910225 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 16 17:10:25 mds1 kernel: LustreError: 138-a: emc-MDT0000: A client on nid 192.168.1.55@o2ib was evicted due to a lock blocking callback time out: rc -107
Jan 16 17:13:36 mds1 kernel: Lustre: MGS: haven't heard from client e72b44b9-20ea-699e-d363-4345641ec44b (at 192.168.1.55@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880831792800, cur 1389910416 expire 1389910266 last 1389910187
Jan 16 17:17:38 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client 4ff2c167-aa19-8225-a595-af18b088d8bc (at 192.168.1.57@o2ib) in 237 seconds. I think it's dead, and I am evicting it. exp ffff88083186f800, cur 1389910658 expire 1389910508 last 1389910421
Jan 17 01:20:54 mds1 init: tty (/dev/tty1) main process (3808) killed by TERM signal
Jan 17 01:20:54 mds1 init: tty (/dev/tty2) main process (3810) killed by TERM signal
Jan 17 01:20:54 mds1 init: tty (/dev/tty3) main process (3812) killed by TERM signal
Jan 17 01:20:54 mds1 init: tty (/dev/tty4) main process (3814) killed by TERM signal
Jan 17 01:20:54 mds1 init: tty (/dev/tty5) main process (3817) killed by TERM signal
Jan 17 01:20:54 mds1 init: tty (/dev/tty6) main process (3819) killed by TERM signal
Jan 17 01:20:55 mds1 abrtd: Got signal 15, exiting
Jan 17 01:20:59 mds1 acpid: exiting
Jan 17 01:20:59 mds1 multipathd: mdt2: event checker exit
Jan 17 01:20:59 mds1 multipathd: dm-2: remove map (uevent)
Jan 17 01:20:59 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 17 01:20:59 mds1 kernel: sd 2:0:0:2: emc: Detached
Jan 17 01:20:59 mds1 kernel: sd 1:0:0:2: emc: Detached
Jan 17 01:20:59 mds1 multipathd: dm-2: remove map (uevent)
Jan 17 01:20:59 mds1 multipathd: dm-2: devmap not registered, can't remove
Jan 17 01:20:59 mds1 multipathd: mgt: stop event checker thread (140402440541952)
Jan 17 01:20:59 mds1 multipathd: mdt1: stop event checker thread (140402429798144)
Jan 17 01:20:59 mds1 multipathd: --------shut down-------
Jan 17 01:20:59 mds1 init: Disconnected from system bus
Jan 17 01:20:59 mds1 console-kit-daemon[89767]: WARNING: no sender#012
Jan 17 01:20:59 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 17 01:20:59 mds1 auditd[3112]: The audit daemon is exiting.
Jan 17 01:20:59 mds1 kernel: type=1305 audit(1389939659.923:1471): audit_pid=0 old=3112 auid=4294967295 ses=4294967295 res=1
Jan 17 01:23:57 mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jan 17 01:23:57 mds1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3141" x-info="http://www.rsyslog.com"] start
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys cpuset
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys cpu
Jan 17 01:23:57 mds1 kernel: Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013
Jan 17 01:23:57 mds1 kernel: Command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 17 01:23:57 mds1 kernel: KERNEL supported cpus:
Jan 17 01:23:57 mds1 kernel:  Intel GenuineIntel
Jan 17 01:23:57 mds1 kernel:  AMD AuthenticAMD
Jan 17 01:23:57 mds1 kernel:  Centaur CentaurHauls
Jan 17 01:23:57 mds1 kernel: BIOS-provided physical RAM map:
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 0000000000100000 - 00000000bb3c7000 (usable)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bb3c7000 - 00000000bdd2f000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bdd2f000 - 00000000bde27000 (ACPI NVS)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bde27000 - 00000000bdea1000 (ACPI data)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bdea1000 - 00000000bdf2f000 (ACPI NVS)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bdf2f000 - 00000000bdfb1000 (ACPI data)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved)
Jan 17 01:23:57 mds1 kernel: BIOS-e820: 0000000100000000 - 0000001040000000 (usable)
Jan 17 01:23:57 mds1 kernel: DMI 2.6 present.
Jan 17 01:23:57 mds1 kernel: SMBIOS version 2.6 @ 0xF0440
Jan 17 01:23:57 mds1 kernel: last_pfn = 0x1040000 max_arch_pfn = 0x400000000
Jan 17 01:23:57 mds1 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jan 17 01:23:57 mds1 kernel: last_pfn = 0xbe000 max_arch_pfn = 0x400000000
Jan 17 01:23:57 mds1 kernel: Using GB pages for direct mapping
Jan 17 01:23:57 mds1 kernel: init_memory_mapping: 0000000000000000-00000000be000000
Jan 17 01:23:57 mds1 kernel: init_memory_mapping: 0000000100000000-0000001040000000
Jan 17 01:23:57 mds1 kernel: RAMDISK: 36c09000 - 37fefa44
Jan 17 01:23:57 mds1 kernel: ACPI: RSDP 00000000000f0410 00024 (v02 INTEL )
Jan 17 01:23:57 mds1 kernel: ACPI: XSDT 00000000bdfaed98 000BC (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: FACP 00000000bdfae918 000F4 (v04 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607)
Jan 17 01:23:57 mds1 kernel: ACPI: DSDT 00000000bdf96018 16780 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: ACPI: FACS 00000000bdfaef40 00040
Jan 17 01:23:57 mds1 kernel: ACPI: APIC 00000000bdfad718 0066A (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: SPMI 00000000bdfb0c18 00040 (v05 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: MCFG 00000000bdfb0b98 0003C (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: WDDT 00000000bdfb0f18 00040 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: SRAT 00000000bdf93e18 00168 (v03 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: SLIT 00000000bdfb0e98 00030 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: MSCT 00000000bdfafd18 00090 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: HPET 00000000bdfb0e18 00038 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: SSDT 00000000bdfb0d18 0002B (v02 INTEL   S2600GL 00001000 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL   S2600GL 00004000 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: ACPI: DMAR 00000000bdfae618 00148 (v01 INTEL   S2600GL 06222004 INTL 20090903)
Jan 17 01:23:57 mds1 kernel: ACPI: HEST 00000000bdf95f18 000A8 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 17 01:23:57 mds1 kernel: ACPI: BERT 00000000bdfb0d98 00030 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 17 01:23:57 mds1 kernel: ACPI: ERST 00000000bdf95c98 00230 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 17 01:23:57 mds1 kernel: ACPI: EINJ 00000000bdfaec18 00130 (v01 INTEL   S2600GL 00000001 INTL 00000001)
Jan 17 01:23:57 mds1 kernel: ACPI: SSDT 00000000bdf94018 00F41 (v02 INTEL   S2600GL 00000002 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: ACPI: SSDT 00000000bdf94f98 00045 (v02 INTEL   S2600GL 00000001 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: ACPI: SSDT 00000000bdf93c18 00181 (v02 INTEL   S2600GL 00000003 INTL 20100331)
Jan 17 01:23:57 mds1 kernel: Setting APIC routing to flat.
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 38 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 40 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: PXM 1 -> APIC 42 -> Node 1
Jan 17 01:23:57 mds1 kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 17 01:23:57 mds1 kernel: SRAT: Node 0 PXM 0 100000000-840000000
Jan 17 01:23:57 mds1 kernel: SRAT: Node 1 PXM 1 840000000-1040000000
Jan 17 01:23:57 mds1 kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 17 01:23:57 mds1 kernel:  NODE_DATA [00000000000090c0 - 000000000003d0bf]
Jan 17 01:23:57 mds1 kernel:  bootmap [0000000000100000 -  0000000000207fff] pages 108
Jan 17 01:23:57 mds1 kernel: (9 early reservations) ==> bootmem [0000000000 - 0840000000]
Jan 17 01:23:57 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Jan 17 01:23:57 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Jan 17 01:23:57 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
Jan 17 01:23:57 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK ==> [0036c09000 - 0037fefa44]
Jan 17 01:23:57 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved ==> [000009ac00 - 0000100000]
Jan 17 01:23:57 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK ==> [000201c000 - 000201c457]
Jan 17 01:23:57 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Jan 17 01:23:57 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Jan 17 01:23:57 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP ==> [0000009000 - 00000090c0]
Jan 17 01:23:57 mds1 kernel: Bootmem setup node 1 0000000840000000-0000001040000000
Jan 17 01:23:57 mds1 kernel:  NODE_DATA [0000000840000040 - 000000084003403f]
Jan 17 01:23:57 mds1 kernel:  bootmap [0000000840035000 -  0000000840134fff] pages 100
Jan 17 01:23:57 mds1 kernel: (9 early reservations) ==> bootmem [0840000000 - 1040000000]
Jan 17 01:23:57 mds1 kernel:  #0 [0000000000 - 0000001000]   BIOS data page
Jan 17 01:23:57 mds1 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE
Jan 17 01:23:57 mds1 kernel:  #2 [0001000000 - 000201b0a4]    TEXT DATA BSS
Jan 17 01:23:57 mds1 kernel:  #3 [0036c09000 - 0037fefa44]          RAMDISK
Jan 17 01:23:57 mds1 kernel:  #4 [000009ac00 - 0000100000]    BIOS reserved
Jan 17 01:23:57 mds1 kernel:  #5 [000201c000 - 000201c457]              BRK
Jan 17 01:23:57 mds1 kernel:  #6 [0000008000 - 0000009000]          PGTABLE
Jan 17 01:23:57 mds1 kernel:  #7 [0000001000 - 0000001030]        ACPI SLIT
Jan 17 01:23:57 mds1 kernel:  #8 [0000009000 - 00000090c0]       MEMNODEMAP
Jan 17 01:23:57 mds1 kernel: found SMP MP-table at [ffff8800000fcd10] fcd10
Jan 17 01:23:57 mds1 kernel: Reserving 133MB of memory at 48MB for crashkernel (System RAM: 66560MB)
Jan 17 01:23:57 mds1 kernel: Zone PFN ranges:
Jan 17 01:23:57 mds1 kernel:  DMA      0x00000001 -> 0x00001000
Jan 17 01:23:57 mds1 kernel:  DMA32    0x00001000 -> 0x00100000
Jan 17 01:23:57 mds1 kernel:  Normal   0x00100000 -> 0x01040000
Jan 17 01:23:57 mds1 kernel: Movable zone start PFN for each node
Jan 17 01:23:57 mds1 kernel: early_node_map[5] active PFN ranges
Jan 17 01:23:57 mds1 kernel:    0: 0x00000001 -> 0x0000009a
Jan 17 01:23:57 mds1 kernel:    0: 0x00000100 -> 0x000bb3c7
Jan 17 01:23:57 mds1 kernel:    0: 0x000bdfb1 -> 0x000be000
Jan 17 01:23:57 mds1 kernel:    0: 0x00100000 -> 0x00840000
Jan 17 01:23:57 mds1 kernel:    1: 0x00840000 -> 0x01040000
Jan 17 01:23:57 mds1 kernel: ACPI: PM-Timer IO Port: 0x408
Jan 17 01:23:57 mds1 kernel: Setting APIC routing to flat.
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x20] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x22] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x24] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x26] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] enabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Jan 17 01:23:57 mds1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 17 01:23:57 mds1 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Jan 17 01:23:57 mds1 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Jan 17 01:23:57 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24])
Jan 17 01:23:57 mds1 kernel: IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
Jan 17 01:23:57 mds1 kernel: ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48])
Jan 17 01:23:57 mds1 kernel: IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
Jan 17 01:23:57 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 17 01:23:57 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 17 01:23:57 mds1 kernel: Using ACPI (MADT) for SMP configuration information
Jan 17 01:23:57 mds1 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Jan 17 01:23:57 mds1 kernel: SMP: Allowing 128 CPUs, 116 hotplug CPUs
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000bb3c7000 - 00000000bdd2f000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000bdd2f000 - 00000000bde27000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000bde27000 - 00000000bdea1000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000bdea1000 - 00000000bdf2f000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000bdf2f000 - 00000000bdfb1000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000be000000 - 00000000d0000000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000
Jan 17 01:23:57 mds1 kernel: PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000
Jan 17 01:23:57 mds1 kernel: Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000)
Jan 17 01:23:57 mds1 kernel: Booting paravirtualized kernel on bare hardware
Jan 17 01:23:57 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
Jan 17 01:23:57 mds1 kernel: PERCPU: Embedded 31 pages/cpu @ffff880045a00000 s94616 r8192 d24168 u131072
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [0] 000 001 002 003 004 005 012 014 016 018 020 022 024 026 028 030 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [1] 006 007 008 009 010 011 013 015 017 019 021 023 025 027 029 031 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Jan 17 01:23:57 mds1 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Jan 17 01:23:57 mds1 kernel: Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16524614
Jan 17 01:23:57 mds1 kernel: Policy zone: Normal
Jan 17 01:23:57 mds1 kernel: Kernel command line: ro root=UUID=72bb1c31-d4b6-4c50-9b0e-1b598ab23934 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Jan 17 01:23:57 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jan 17 01:23:57 mds1 kernel: xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Jan 17 01:23:57 mds1 kernel: Checking aperture...
Jan 17 01:23:57 mds1 kernel: No AGP bridge found
Jan 17 01:23:57 mds1 kernel: dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
Jan 17 01:23:57 mds1 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 17 01:23:57 mds1 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Jan 17 01:23:57 mds1 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Jan 17 01:23:57 mds1 kernel: Memory: 65857328k/68157440k available (5224k kernel code, 1126724k absent, 1173388k reserved, 7118k data, 1264k init)
Jan 17 01:23:57 mds1 kernel: Hierarchical RCU implementation.
Jan 17 01:23:57 mds1 kernel: NR_IRQS:33024 nr_irqs:2248
Jan 17 01:23:57 mds1 kernel: Extended CMOS year: 2000
Jan 17 01:23:57 mds1 kernel: Console: colour VGA+ 80x25
Jan 17 01:23:57 mds1 kernel: console [tty0] enabled
Jan 17 01:23:57 mds1 kernel: allocated 268435456 bytes of page_cgroup
Jan 17 01:23:57 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jan 17 01:23:57 mds1 kernel: Fast TSC calibration using PIT
Jan 17 01:23:57 mds1 kernel: Detected 1995.122 MHz processor.
Jan 17 01:23:57 mds1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.24 BogoMIPS (lpj=1995122)
Jan 17 01:23:57 mds1 kernel: pid_max: default: 131072 minimum: 1024
Jan 17 01:23:57 mds1 kernel: Security Framework initialized
Jan 17 01:23:57 mds1 kernel: SELinux:  Initializing.
Jan 17 01:23:57 mds1 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jan 17 01:23:57 mds1 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jan 17 01:23:57 mds1 kernel: Mount-cache hash table entries: 256
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys ns
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys cpuacct
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys memory
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys devices
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys freezer
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys net_cls
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys blkio
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys perf_event
Jan 17 01:23:57 mds1 kernel: Initializing cgroup subsys net_prio
Jan 17 01:23:57 mds1 kernel: CPU: Physical Processor ID: 0
Jan 17 01:23:57 mds1 kernel: CPU: Processor Core ID: 0
Jan 17 01:23:57 mds1 kernel: mce: CPU supports 18 MCE banks
Jan 17 01:23:57 mds1 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 17 01:23:57 mds1 kernel: using mwait in idle threads.
Jan 17 01:23:57 mds1 kernel: ACPI: Core revision 20090903
Jan 17 01:23:57 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jan 17 01:23:57 mds1 kernel: ftrace: allocating 21445 entries in 85 pages
Jan 17 01:23:57 mds1 kernel: dmar: Host address width 46
Jan 17 01:23:57 mds1 kernel: dmar: DRHD base: 0x000000fbffe000 flags: 0x0
Jan 17 01:23:57 mds1 kernel: dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 17 01:23:57 mds1 kernel: dmar: DRHD base: 0x000000ebffc000 flags: 0x1
Jan 17 01:23:57 mds1 kernel: dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
Jan 17 01:23:57 mds1 kernel: dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff
Jan 17 01:23:57 mds1 kernel: dmar: ATSR flags: 0x0
Jan 17 01:23:57 mds1 kernel: IOAPIC id 2 under DRHD base 0xfbffe000
Jan 17 01:23:57 mds1 kernel: IOAPIC id 0 under DRHD base 0xebffc000
Jan 17 01:23:57 mds1 kernel: IOAPIC id 1 under DRHD base 0xebffc000
Jan 17 01:23:57 mds1 kernel: HPET id 0 under DRHD base 0xebffc000
Jan 17 01:23:57 mds1 kernel: Enabled IRQ remapping in x2apic mode
Jan 17 01:23:57 mds1 kernel: Enabling x2apic
Jan 17 01:23:57 mds1 kernel: Enabled x2apic
Jan 17 01:23:57 mds1 kernel: APIC routing finalized to cluster x2apic.
Jan 17 01:23:57 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Jan 17 01:23:57 mds1 kernel: CPU0: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping 07
Jan 17 01:23:57 mds1 kernel: Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
Jan 17 01:23:57 mds1 kernel: PEBS disabled due to CPU errata.
Jan 17 01:23:57 mds1 kernel: ... version:                3
Jan 17 01:23:57 mds1 kernel: ... bit width:              48
Jan 17 01:23:57 mds1 kernel: ... generic registers:      8
Jan 17 01:23:57 mds1 kernel: ... value mask:             0000ffffffffffff
Jan 17 01:23:57 mds1 kernel: ... max period:             000000007fffffff
Jan 17 01:23:57 mds1 kernel: ... fixed-purpose events:   3
Jan 17 01:23:57 mds1 kernel: ... event mask:             00000007000000ff
Jan 17 01:23:57 mds1 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jan 17 01:23:57 mds1 kernel: Booting Node   0, Processors  #1 #2 #3 #4 #5 Ok.
Jan 17 01:23:57 mds1 kernel: Booting Node   1, Processors  #6 #7 #8 #9 #10 #11
Jan 17 01:23:57 mds1 kernel: Brought up 12 CPUs
Jan 17 01:23:57 mds1 kernel: Total of 12 processors activated (47880.54 BogoMIPS).
Jan 17 01:23:57 mds1 kernel: devtmpfs: initialized
Jan 17 01:23:57 mds1 kernel: PM: Registering ACPI NVS region at bdd2f000 (1015808 bytes)
Jan 17 01:23:57 mds1 kernel: PM: Registering ACPI NVS region at bdea1000 (581632 bytes)
Jan 17 01:23:57 mds1 kernel: regulator: core version 0.5
Jan 17 01:23:57 mds1 kernel: NET: Registered protocol family 16
Jan 17 01:23:57 mds1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jan 17 01:23:57 mds1 kernel: ACPI: bus type pci registered
Jan 17 01:23:57 mds1 kernel: PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255
Jan 17 01:23:57 mds1 kernel: PCI: MCFG area at c0000000 reserved in E820
Jan 17 01:23:57 mds1 kernel: PCI: Using MMCONFIG at c0000000 - cfffffff
Jan 17 01:23:57 mds1 kernel: PCI: Using configuration type 1 for base access
Jan 17 01:23:57 mds1 kernel: bio: create slab <bio-0> at 0
Jan 17 01:23:57 mds1 kernel: ACPI: Executed 1 blocks of module-level executable AML code
Jan 17 01:23:57 mds1 kernel: ACPI: BIOS _OSI(Linux) query ignored
Jan 17 01:23:57 mds1 kernel: ACPI: Interpreter enabled
Jan 17 01:23:57 mds1 kernel: ACPI: (supports S0 S1 S5)
Jan 17 01:23:57 mds1 kernel: ACPI: Using IOAPIC for interrupt routing
Jan 17 01:23:57 mds1 kernel: ACPI: No dock devices found.
Jan 17 01:23:57 mds1 kernel: HEST: Table parsing has been initialized.
Jan 17 01:23:57 mds1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [io  0x0000-0xbfff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cc000-0x000d33ff])
Jan 17 01:23:57 mds1 kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:00.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:16.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:16.1: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1a.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1d.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1f.2: PME# supported from D3hot
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1f.2: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a] (subtractive decode)
Jan 17 01:23:57 mds1 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jan 17 01:23:57 mds1 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0x03b0-0x03df]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:01: host bridge window [io  0xc000-0xffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Jan 17 01:23:57 mds1 kernel: pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PME# disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 17 01:23:57 mds1 kernel: pci0000:80: Requesting ACPI _OSC control (0x1d)
Jan 17 01:23:57 mds1 kernel: pci0000:80: ACPI _OSC control (0x15) granted
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
Jan 17 01:23:57 mds1 kernel: pci0000:7f: Requesting ACPI _OSC control (0x1d)
Jan 17 01:23:57 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
Jan 17 01:23:57 mds1 kernel: pci0000:ff: Requesting ACPI _OSC control (0x1d)
Jan 17 01:23:57 mds1 kernel: Unable to assume _OSC PCIe control. Disabling ASPM
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 17 01:23:57 mds1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
Jan 17 01:23:57 mds1 kernel: vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none
Jan 17 01:23:57 mds1 kernel: vgaarb: loaded
Jan 17 01:23:57 mds1 kernel: vgaarb: bridge control possible 0000:09:00.0
Jan 17 01:23:57 mds1 kernel: SCSI subsystem initialized
Jan 17 01:23:57 mds1 kernel: usbcore: registered new interface driver usbfs
Jan 17 01:23:57 mds1 kernel: usbcore: registered new interface driver hub
Jan 17 01:23:57 mds1 kernel: usbcore: registered new device driver usb
Jan 17 01:23:57 mds1 kernel: PCI: Using ACPI for IRQ routing
Jan 17 01:23:57 mds1 kernel: NetLabel: Initializing
Jan 17 01:23:57 mds1 kernel: NetLabel:  domain hash size = 128
Jan 17 01:23:57 mds1 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jan 17 01:23:57 mds1 kernel: NetLabel:  unlabeled traffic allowed by default
Jan 17 01:23:57 mds1 kernel: HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Jan 17 01:23:57 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 74, 75, 76, 77, 78, 0
Jan 17 01:23:57 mds1 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Jan 17 01:23:57 mds1 kernel: Switching to clocksource hpet
Jan 17 01:23:57 mds1 kernel: pnp: PnP ACPI init
Jan 17 01:23:57 mds1 kernel: ACPI: bus type pnp registered
Jan 17 01:23:57 mds1 kernel: pnp: PnP ACPI: found 17 devices
Jan 17 01:23:57 mds1 kernel: ACPI: ACPI bus type pnp unregistered
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0680-0x069f] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0xffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0400-0x0453] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0458-0x047f] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0500-0x057f] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0600-0x061f] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0ca2-0x0ca5] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:05: [io  0x0cf9] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:07: [io  0x0454-0x0457] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:0a: [mem 0xd1d70000-0xd1d70fff] has been reserved
Jan 17 01:23:57 mds1 kernel: system 00:0b: [mem 0xebffc000-0xebffdfff] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:0d: [mem 0xfbffe000-0xfbffffff] could not be reserved
Jan 17 01:23:57 mds1 kernel: system 00:0e: [mem 0x00000000-0x0009cfff] could not be reserved
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: BAR 7: assigned [mem 0xd19b4000-0xd19d3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: BAR 7: set to [mem 0xd19b4000-0xd19d3fff] (PCI address [0xd19b4000-0xd19d3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: BAR 10: assigned [mem 0xd19d4000-0xd19f3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.0: BAR 10: set to [mem 0xd19d4000-0xd19f3fff] (PCI address [0xd19d4000-0xd19f3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: BAR 7: assigned [mem 0xd19f4000-0xd1a13fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: BAR 7: set to [mem 0xd19f4000-0xd1a13fff] (PCI address [0xd19f4000-0xd1a13fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: BAR 10: assigned [mem 0xd1a14000-0xd1a33fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.1: BAR 10: set to [mem 0xd1a14000-0xd1a33fff] (PCI address [0xd1a14000-0xd1a33fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: BAR 7: assigned [mem 0xd1a34000-0xd1a53fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: BAR 7: set to [mem 0xd1a34000-0xd1a53fff] (PCI address [0xd1a34000-0xd1a53fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: BAR 10: assigned [mem 0xd1a54000-0xd1a73fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.2: BAR 10: set to [mem 0xd1a54000-0xd1a73fff] (PCI address [0xd1a54000-0xd1a73fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: BAR 7: assigned [mem 0xd1a74000-0xd1a93fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: BAR 7: set to [mem 0xd1a74000-0xd1a93fff] (PCI address [0xd1a74000-0xd1a93fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: BAR 10: assigned [mem 0xd1a94000-0xd1ab3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:02:00.3: BAR 10: set to [mem 0xd1a94000-0xd1ab3fff] (PCI address [0xd1a94000-0xd1ab3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PCI bridge to [bus 02-03]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1:   bridge window [mem 0xd1900000-0xd1afffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PCI bridge to [bus 05-05]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2:   bridge window [io  0x2000-0x2fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2:   bridge window [mem 0xd1c00000-0xd1cfffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PCI bridge to [bus 06-06]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0:   bridge window [mem 0xd1b00000-0xd1bfffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PCI bridge to [bus 07-07]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PCI bridge to [bus 08-08]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PCI bridge to [bus 09-09]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd18fffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0: PCI bridge to [bus 0a-0a]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1e.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PCI bridge to [bus 81-81]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PCI bridge to [bus 82-82]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xec000000-0xec0fffff]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0:   bridge window [mem 0xfb000000-0xfb7fffff 64bit pref]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PCI bridge to [bus 83-83]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PCI bridge to [bus 84-84]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2:   bridge window [io  disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2:   bridge window [mem disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2:   bridge window [mem pref disabled]
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 17 01:23:57 mds1 kernel: pci 0000:00:01.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 17 01:23:57 mds1 kernel: pci 0000:00:02.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47
Jan 17 01:23:57 mds1 kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 17 01:23:57 mds1 kernel: pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jan 17 01:23:57 mds1 kernel: pci 0000:80:01.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 17 01:23:57 mds1 kernel: pci 0000:80:02.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 17 01:23:57 mds1 kernel: pci 0000:80:03.2: PCI INT A -> GSI 71 (level, low) -> IRQ 71
Jan 17 01:23:57 mds1 kernel: NET: Registered protocol family 2
Jan 17 01:23:57 mds1 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jan 17 01:23:57 mds1 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jan 17 01:23:57 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jan 17 01:23:57 mds1 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jan 17 01:23:57 mds1 kernel: TCP reno registered
Jan 17 01:23:57 mds1 kernel: NET: Registered protocol family 1
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1a.0: PCI INT B disabled
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 17 01:23:57 mds1 kernel: pci 0000:00:1d.0: PCI INT A disabled
Jan 17 01:23:57 mds1 kernel: Trying to unpack rootfs image as initramfs...
Jan 17 01:23:57 mds1 kernel: Freeing initrd memory: 20378k freed
Jan 17 01:23:57 mds1 kernel: dmar: Unsupported device scope
Jan 17 01:23:57 mds1 kernel: audit: initializing netlink socket (disabled)
Jan 17 01:23:57 mds1 kernel: type=2000 audit(1389939815.056:1): initialized
Jan 17 01:23:57 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jan 17 01:23:57 mds1 kernel: VFS: Disk quotas dquot_6.5.2
Jan 17 01:23:57 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 17 01:23:57 mds1 kernel: msgmni has been set to 32768
Jan 17 01:23:57 mds1 kernel: alg: No test for stdrng (krng)
Jan 17 01:23:57 mds1 kernel: ksign: Installing public key data
Jan 17 01:23:57 mds1 kernel: Loading keyring
Jan 17 01:23:57 mds1 kernel: - Added public key 102B0AC9EF026FEE
Jan 17 01:23:57 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jan 17 01:23:57 mds1 kernel: - Added public key D4A26C9CCD09BEDA
Jan 17 01:23:57 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jan 17 01:23:57 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Jan 17 01:23:57 mds1 kernel: io scheduler noop registered
Jan 17 01:23:57 mds1 kernel: io scheduler anticipatory registered
Jan 17 01:23:57 mds1 kernel: io scheduler deadline registered (default)
Jan 17 01:23:57 mds1 kernel: io scheduler cfq registered
Jan 17 01:23:57 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jan 17 01:23:57 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jan 17 01:23:57 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jan 17 01:23:57 mds1 kernel: acpiphp: Slot [2] registered
Jan 17 01:23:57 mds1 kernel: acpiphp: Slot [2-1] registered
Jan 17 01:23:57 mds1 kernel: acpiphp: Slot [2-2] registered
Jan 17 01:23:57 mds1 kernel: acpiphp: Slot [2-3] registered
Jan 17 01:23:57 mds1 kernel: ipmi message handler version 39.2
Jan 17 01:23:57 mds1 kernel: IPMI System Interface driver.
Jan 17 01:23:57 mds1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan 17 01:23:57 mds1 kernel: ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Jan 17 01:23:57 mds1 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan 17 01:23:57 mds1 kernel: ipmi: Found new BMC (man_id: 0x000157,  prod_id: 0x0049, dev_id: 0x21)
Jan 17 01:23:57 mds1 kernel: IPMI kcs interface initialized
Jan 17 01:23:57 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Jan 17 01:23:57 mds1 kernel: ACPI: Power Button [PWRF]
Jan 17 01:23:57 mds1 kernel: ERST: Could not register with persistent store
Jan 17 01:23:57 mds1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.
Jan 17 01:23:57 mds1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
Jan 17 01:23:57 mds1 kernel: Non-volatile memory driver v1.3
Jan 17 01:23:57 mds1 kernel: Linux agpgart interface v0.103
Jan 17 01:23:57 mds1 kernel: crash memory driver: version 1.1
Jan 17 01:23:57 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jan 17 01:23:57 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 17 01:23:57 mds1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 17 01:23:57 mds1 kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jan 17 01:23:57 mds1 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jan 17 01:23:57 mds1 kernel: brd: module loaded
Jan 17 01:23:57 mds1 kernel: loop: module loaded
Jan 17 01:23:57 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Jan 17 01:23:57 mds1 kernel: Fixed MDIO Bus: probed
Jan 17 01:23:57 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: EHCI Host Controller
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: debug port 2
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd1d20000
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Jan 17 01:23:57 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jan 17 01:23:57 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 17 01:23:57 mds1 kernel: usb usb1: Product: EHCI Host Controller
Jan 17 01:23:57 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 17 01:23:57 mds1 kernel: usb usb1: SerialNumber: 0000:00:1a.0
Jan 17 01:23:57 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: hub 1-0:1.0: USB hub found
Jan 17 01:23:57 mds1 kernel: hub 1-0:1.0: 2 ports detected
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: EHCI Host Controller
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: debug port 2
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd1d10000
Jan 17 01:23:57 mds1 kernel: ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Jan 17 01:23:57 mds1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jan 17 01:23:57 mds1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jan 17 01:23:57 mds1 kernel: usb usb2: Product: EHCI Host Controller
Jan 17 01:23:57 mds1 kernel: usb usb2: Manufacturer: Linux 2.6.32-358.18.1.el6_lustre.x86_64 ehci_hcd
Jan 17 01:23:57 mds1 kernel: usb usb2: SerialNumber: 0000:00:1d.0
Jan 17 01:23:57 mds1 kernel: usb usb2: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: hub 2-0:1.0: USB hub found
Jan 17 01:23:57 mds1 kernel: hub 2-0:1.0: 2 ports detected
Jan 17 01:23:57 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jan 17 01:23:57 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jan 17 01:23:57 mds1 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jan 17 01:23:57 mds1 kernel: Refined TSC clocksource calibration: 1995.192 MHz.
Jan 17 01:23:57 mds1 kernel: Switching to clocksource tsc
Jan 17 01:23:57 mds1 kernel: i8042.c: Can't read CTR while initializing i8042.
Jan 17 01:23:57 mds1 kernel: i8042: probe of i8042 failed with error -5
Jan 17 01:23:57 mds1 kernel: usb 1-1: new high speed USB device number 2 using ehci_hcd
Jan 17 01:23:57 mds1 kernel: mice: PS/2 mouse device common for all mice
Jan 17 01:23:57 mds1 kernel: rtc_cmos 00:06: RTC can wake from S4
Jan 17 01:23:57 mds1 kernel: rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
Jan 17 01:23:57 mds1 kernel: rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Jan 17 01:23:57 mds1 kernel: cpuidle: using governor ladder
Jan 17 01:23:57 mds1 kernel: cpuidle: using governor menu
Jan 17 01:23:57 mds1 kernel: EFI Variables Facility v0.08 2004-May-17
Jan 17 01:23:57 mds1 kernel: usbcore: registered new interface driver hiddev
Jan 17 01:23:57 mds1 kernel: usbcore: registered new interface driver usbhid
Jan 17 01:23:57 mds1 kernel: usbhid: v2.6:USB HID core driver
Jan 17 01:23:57 mds1 kernel: TCP cubic registered
Jan 17 01:23:57 mds1 kernel: Initializing XFRM netlink socket
Jan 17 01:23:57 mds1 kernel: NET: Registered protocol family 17
Jan 17 01:23:57 mds1 kernel: registered taskstats version 1
Jan 17 01:23:57 mds1 kernel: rtc_cmos 00:06: setting system clock to 2014-01-17 06:23:37 UTC (1389939817)
Jan 17 01:23:57 mds1 kernel: Initalizing network drop monitor service
Jan 17 01:23:57 mds1 kernel: Freeing unused kernel memory: 1264k freed
Jan 17 01:23:57 mds1 kernel: Write protecting the kernel read-only data: 10240k
Jan 17 01:23:57 mds1 kernel: Freeing unused kernel memory: 900k freed
Jan 17 01:23:57 mds1 kernel: Freeing unused kernel memory: 1672k freed
Jan 17 01:23:57 mds1 kernel: dracut: dracut-004-303.el6
Jan 17 01:23:57 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jan 17 01:23:57 mds1 kernel: dracut: rd_NO_LVM: removing LVM activation
Jan 17 01:23:57 mds1 kernel: device-mapper: uevent: version 1.0.3
Jan 17 01:23:57 mds1 kernel: device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Jan 17 01:23:57 mds1 kernel: udev: starting version 147
Jan 17 01:23:57 mds1 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Jan 17 01:23:57 mds1 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 17 01:23:57 mds1 kernel: usb 1-1: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: hub 1-1:1.0: USB hub found
Jan 17 01:23:57 mds1 kernel: hub 1-1:1.0: 6 ports detected
Jan 17 01:23:57 mds1 kernel: dracut: Starting plymouth daemon
Jan 17 01:23:57 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jan 17 01:23:57 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jan 17 01:23:57 mds1 kernel: usb 2-1: new high speed USB device number 2 using ehci_hcd
Jan 17 01:23:57 mds1 kernel: ACPI: WMI: Mapper loaded
Jan 17 01:23:57 mds1 kernel: megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Jan 17 01:23:57 mds1 kernel: megasas: 0x1000:0x005b:0x8086:0x3513: bus 5:slot 0:func 0
Jan 17 01:23:57 mds1 kernel: megaraid_sas 0000:05:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34
Jan 17 01:23:57 mds1 kernel: megasas: FW now in Ready state
Jan 17 01:23:57 mds1 kernel: megasas:IOC Init cmd success
Jan 17 01:23:57 mds1 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Jan 17 01:23:57 mds1 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jan 17 01:23:57 mds1 kernel: usb 2-1: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: hub 2-1:1.0: USB hub found
Jan 17 01:23:57 mds1 kernel: hub 2-1:1.0: 8 ports detected
Jan 17 01:23:57 mds1 kernel: megasas: INIT adapter done
Jan 17 01:23:57 mds1 kernel: scsi0 : LSI SAS based MegaRAID driver
Jan 17 01:23:57 mds1 kernel: scsi 0:0:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 17 01:23:57 mds1 kernel: scsi 0:2:0:0: Direct-Access     Intel    RMS25CB080       3.19 PQ: 0 ANSI: 5
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.08.06.4-k.
Jan 17 01:23:57 mds1 kernel: qla2xxx 0000:06:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.0]-001d: Found an ISP2532 irq 40 iobase 0xffffc90016e4c000.
Jan 17 01:23:57 mds1 kernel: qla2xxx 0000:06:00.0: firmware: requesting ql2500_fw.bin
Jan 17 01:23:57 mds1 kernel: scsi1 : qla2xxx
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.0]-00fb:1: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.0]-00fc:1: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.0 hdma+ host#=1 fw=5.08.00 (90d5).
Jan 17 01:23:57 mds1 kernel: qla2xxx 0000:06:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.1]-001d: Found an ISP2532 irq 44 iobase 0xffffc90016e34000.
Jan 17 01:23:57 mds1 kernel: scsi2 : qla2xxx
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.1]-00fb:2: QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA.
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.1]-00fc:2: ISP2532: PCIe (5.0GT/s x4) @ 0000:06:00.1 hdma+ host#=2 fw=5.08.00 (90d5).
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: new low speed USB device number 3 using ehci_hcd
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: New USB device found, idVendor=0624, idProduct=0307
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: Product: Avocent DSRIQ-USB
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: Manufacturer: Avocent
Jan 17 01:23:57 mds1 kernel: usb 2-1.2: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/input/input2
Jan 17 01:23:57 mds1 kernel: generic-usb 0003:0624:0307.0001: input,hidraw0: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input0
Jan 17 01:23:57 mds1 kernel: input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.1/input/input3
Jan 17 01:23:57 mds1 kernel: generic-usb 0003:0624:0307.0002: input,hidraw1: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.0-1.2/input1
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: new full speed USB device number 4 using ehci_hcd
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.0]-505f:1: Link is operational (8 Gbps).
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: Product: Virtual Keyboard and Mouse
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: Manufacturer: American Megatrends Inc.
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: SerialNumber: serial
Jan 17 01:23:57 mds1 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Jan 17 01:23:57 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
Jan 17 01:23:57 mds1 kernel: generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
Jan 17 01:23:57 mds1 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input5
Jan 17 01:23:57 mds1 kernel: generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.1]-505f:2: Link is operational (8 Gbps).
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.0]-1020:1: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2dce, mb[3]=a100, cmd=6a ****.
Jan 17 01:23:57 mds1 kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: qla2xxx [0000:06:00.1]-1020:2: **** Failed mbx[0]=4006, mb[1]=7fe, mb[2]=2de9, mb[3]=f100, cmd=6a ****.
Jan 17 01:23:57 mds1 kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 1           0533 PQ: 0 ANSI: 4
Jan 17 01:23:57 mds1 kernel: ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jan 17 01:23:57 mds1 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Jan 17 01:23:57 mds1 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Jan 17 01:23:57 mds1 kernel: scsi3 : ahci
Jan 17 01:23:57 mds1 kernel: scsi4 : ahci
Jan 17 01:23:57 mds1 kernel: scsi5 : ahci
Jan 17 01:23:57 mds1 kernel: scsi6 : ahci
Jan 17 01:23:57 mds1 kernel: scsi7 : ahci
Jan 17 01:23:57 mds1 kernel: scsi8 : ahci
Jan 17 01:23:57 mds1 kernel: ata1: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00100 irq 107
Jan 17 01:23:57 mds1 kernel: ata2: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00180 irq 107
Jan 17 01:23:57 mds1 kernel: ata3: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00200 irq 107
Jan 17 01:23:57 mds1 kernel: ata4: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00280 irq 107
Jan 17 01:23:57 mds1 kernel: ata5: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00300 irq 107
Jan 17 01:23:57 mds1 kernel: ata6: SATA max UDMA/133 abar m2048@0xd1d00000 port 0xd1d00380 irq 107
Jan 17 01:23:57 mds1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata1.00: ATAPI: DV-28S-Y, 1.4A, max UDMA/100, ATAPI AN
Jan 17 01:23:57 mds1 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jan 17 01:23:57 mds1 kernel: ata1.00: configured for UDMA/100
Jan 17 01:23:57 mds1 kernel: scsi 3:0:0:0: CD-ROM            TEAC     DV-28S-Y         1.4A PQ: 0 ANSI: 5
Jan 17 01:23:57 mds1 kernel: sd 0:2:0:0: [sda] 1755858944 512-byte logical blocks: (898 GB/837 GiB)
Jan 17 01:23:57 mds1 kernel: sd 0:2:0:0: [sda] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: [sdb] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: [sdd] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: [sdc] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: [sdc] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: [sdd] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: [sde] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: [sdf] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sda:
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: [sde] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: [sdf] Write Protect is off
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: [sdg] 1125676544 512-byte logical blocks: (576 GB/536 GiB)
Jan 17 01:23:57 mds1 kernel: sdb:
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: [sdg] Write Protect is off
Jan 17 01:23:57 mds1 kernel: unknown partition table
Jan 17 01:23:57 mds1 kernel: sdc:
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 01:23:57 mds1 kernel: sde:
Jan 17 01:23:57 mds1 kernel: sdd: unknown partition table
Jan 17 01:23:57 mds1 kernel: sdf:
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: unknown partition table
Jan 17 01:23:57 mds1 kernel: unknown partition table
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: [sdc] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: sdg: unknown partition table
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: [sdd] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: unknown partition table
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: [sdf] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: [sdg] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: sda1 sda2 sda3
Jan 17 01:23:57 mds1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Jan 17 01:23:57 mds1 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jan 17 01:23:57 mds1 kernel: Uniform CD-ROM driver Revision: 3.20
Jan 17 01:23:57 mds1 kernel: EXT4-fs (sda2): INFO: recovery required on readonly filesystem
Jan 17 01:23:57 mds1 kernel: EXT4-fs (sda2): write access will be enabled during recovery
Jan 17 01:23:57 mds1 kernel: EXT4-fs (sda2): recovery complete
Jan 17 01:23:57 mds1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jan 17 01:23:57 mds1 kernel: dracut: Mounted root filesystem /dev/sda2
Jan 17 01:23:57 mds1 kernel: SELinux:  Disabled at runtime.
Jan 17 01:23:57 mds1 kernel: type=1404 audit(1389939821.245:2): selinux=0 auid=4294967295 ses=4294967295
Jan 17 01:23:57 mds1 kernel: dracut: 
Jan 17 01:23:57 mds1 kernel: dracut: Switching root
Jan 17 01:23:57 mds1 kernel: udev: starting version 147
Jan 17 01:23:57 mds1 kernel: pps_core: LinuxPPS API ver. 1 registered
Jan 17 01:23:57 mds1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Jan 17 01:23:57 mds1 kernel: PTP clock support registered
Jan 17 01:23:57 mds1 kernel: dca service started, version 1.12.1
Jan 17 01:23:57 mds1 kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Jan 17 01:23:57 mds1 kernel: igb: Copyright (c) 2007-2012 Intel Corporation.
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: PCI INT A -> GSI 27 (level, low) -> IRQ 27
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: added PHC on eth0
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d5
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: eth0: PBA No: 100000-000
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: PCI INT B -> GSI 30 (level, low) -> IRQ 30
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: added PHC on eth1
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d6
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: eth1: PBA No: 100000-000
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: PCI INT C -> GSI 28 (level, low) -> IRQ 28
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: added PHC on eth2
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d7
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: eth2: PBA No: 100000-000
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: PCI INT D -> GSI 29 (level, low) -> IRQ 29
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: added PHC on eth3
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:67:be:d8
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: eth3: PBA No: 100000-000
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Jan 17 01:23:57 mds1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: Attached scsi generic sg2 type 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: Attached scsi generic sg3 type 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: Attached scsi generic sg5 type 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: Attached scsi generic sg6 type 0
Jan 17 01:23:57 mds1 kernel: sr 3:0:0:0: Attached scsi generic sg7 type 5
Jan 17 01:23:57 mds1 kernel: ioatdma: Intel(R) QuickData Technology Driver 4.00
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.0: DCA enabled
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.1: DCA enabled
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.2: DCA enabled
Jan 17 01:23:57 mds1 kernel: igb 0000:02:00.3: DCA enabled
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
Jan 17 01:23:57 mds1 kernel: ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
Jan 17 01:23:57 mds1 kernel: iTCO_vendor_support: vendor-support=0
Jan 17 01:23:57 mds1 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Jan 17 01:23:57 mds1 kernel: iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Jan 17 01:23:57 mds1 kernel: i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23
Jan 17 01:23:57 mds1 kernel: ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Jan 17 01:23:57 mds1 kernel: ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Jan 17 01:23:57 mds1 kernel: EDAC MC: Ver: 2.1.0 Oct 11 2013
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
Jan 17 01:23:57 mds1 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
Jan 17 01:23:57 mds1 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
Jan 17 01:23:57 mds1 kernel: EDAC sbridge: Driver loaded.
Jan 17 01:23:57 mds1 kernel: microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x708
Jan 17 01:23:57 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-2d-07
Jan 17 01:23:57 mds1 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jan 17 01:23:57 mds1 kernel: microcode: CPU0 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU1 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU2 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU3 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU4 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU5 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU6 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU7 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU8 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU9 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU10 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: microcode: CPU11 updated to revision 0x70d, date = 2012-05-22 
Jan 17 01:23:57 mds1 kernel: knem 1.0.90mlnx2: initialized
Jan 17 01:23:57 mds1 kernel: device-mapper: multipath: version 1.5.0 loaded
Jan 17 01:23:57 mds1 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jan 17 01:23:57 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 17 01:23:57 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 17 01:23:57 mds1 kernel: device-mapper: table: 253:0: multipath: error getting device
Jan 17 01:23:57 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 17 01:23:57 mds1 kernel: emc: device handler registered
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: connected to SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:0: emc: connected to SP B Port 8 (bound, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: connected to SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:1: emc: connected to SP B Port 8 (bound, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: connected to SP B Port 8 (owned, default SP B)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: emc: detected Clariion VNX7600, flags 0
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:2: emc: connected to SP A Port 8 (bound, default SP B)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Jan 17 01:23:57 mds1 kernel: Adding 4194296k swap on /dev/sda3.  Priority:-1 extents:1 across:4194296k 
Jan 17 01:23:57 mds1 kernel: Compat-mlnx-ofed backport release: 3a2d7bf
Jan 17 01:23:57 mds1 kernel: Backport based on /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git 3a2d7bf
Jan 17 01:23:57 mds1 kernel: compat.git: /mswg/git/mlnx_ofed_2_0/mofed-linux-2.0.git
Jan 17 01:23:57 mds1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Jan 17 01:23:57 mds1 kernel: mlx4_core: Initializing 0000:82:00.0
Jan 17 01:23:57 mds1 kernel: mlx4_core 0000:82:00.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Jan 17 01:23:57 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10296e3000, in_mod=0x1, op_mod=0x0, fw status = 0x40
Jan 17 01:23:57 mds1 kernel: mlx4_core 0000:82:00.0: command SET_PORT (0xc) failed: in_param=0x10296e3000, in_mod=0x2, op_mod=0x0, fw status = 0x40
Jan 17 01:23:57 mds1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Jan 17 01:23:57 mds1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Jan 17 01:23:57 mds1 kernel: NET: Registered protocol family 10
Jan 17 01:23:57 mds1 kernel: lo: Disabled Privacy Extensions
Jan 17 01:23:57 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 17 01:23:57 mds1 kernel: Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16
Jan 17 01:23:57 mds1 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jan 17 01:23:57 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 17 01:23:57 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 17 01:23:57 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 17 01:23:57 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:1: emc: at SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 1:0:0:0: emc: at SP A Port 8 (owned, default SP A)
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: ALUA failover mode detected
Jan 17 01:23:57 mds1 kernel: sd 2:0:0:2: emc: at SP B Port 8 (owned, default SP B)
Jan 17 01:23:57 mds1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jan 17 01:23:57 mds1 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 17 01:23:57 mds1 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jan 17 01:23:57 mds1 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jan 17 01:23:58 mds1 rpc.statd[3236]: Version 1.2.3 starting
Jan 17 01:23:58 mds1 sm-notify[3237]: Version 1.2.3 starting
Jan 17 01:23:58 mds1 kernel: RPC: Registered named UNIX socket transport module.
Jan 17 01:23:58 mds1 kernel: RPC: Registered udp transport module.
Jan 17 01:23:58 mds1 kernel: RPC: Registered tcp transport module.
Jan 17 01:23:58 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jan 17 01:23:58 mds1 kernel: LNet: HW CPU cores: 12, npartitions: 4
Jan 17 01:23:58 mds1 kernel: alg: No test for crc32 (crc32-table)
Jan 17 01:23:58 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Jan 17 01:23:58 mds1 kernel: alg: No test for crc32 (crc32-pclmul)
Jan 17 01:24:02 mds1 kernel: padlock: VIA PadLock Hash Engine not detected.
Jan 17 01:24:02 mds1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-358.18.1.el6_lustre.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Jan 17 01:24:06 mds1 kernel: Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64
Jan 17 01:24:06 mds1 kernel: LNet: Added LNI 192.168.1.40@o2ib [8/256/0/180]
Jan 17 01:24:07 mds1 stonith-ng[3380]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 cib[3379]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 attrd[3382]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 pengine[3383]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 lrmd[3381]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 crmd[3384]:   notice: crm_add_logfile: Additional logging available in /var/log/corosync.log
Jan 17 01:24:07 mds1 logger[3393]: openibd: Set node_desc for mlx4_0: mds1 HCA-1
Jan 17 01:24:09 mds1 kdump: kexec: loaded kdump kernel
Jan 17 01:24:09 mds1 kdump: started up
Jan 17 01:24:09 mds1 ypbind: NIS domain: emc-lustre, NIS server: cn1
Jan 17 01:24:10 mds1 acpid: starting up
Jan 17 01:24:10 mds1 acpid: 1 rule loaded
Jan 17 01:24:10 mds1 acpid: waiting for events: event logging is off
Jan 17 01:24:10 mds1 acpid: client connected from 3596[68:68]
Jan 17 01:24:10 mds1 acpid: 1 client rule loaded
Jan 17 01:24:10 mds1 kernel: ipmi device interface
Jan 17 01:24:11 mds1 automount[3635]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 17 01:24:12 mds1 abrtd: Init complete, entering main loop
Jan 17 01:24:13 mds1 cimserver[3786]: cimserver not started: SSL Exception: Could not access server certificate in /etc/Pegasus/server.pem.
Jan 17 01:24:18 mds1 kernel: mlx4_core 0000:82:00.0: mlx4_ib: Port 1 logical link is up
Jan 17 01:24:20 mds1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jan 17 01:24:51 mds1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 17 01:24:51 mds1 kernel: 
Jan 17 01:24:51 mds1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 17 01:24:51 mds1 kernel: 
Jan 17 01:24:51 mds1 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jan 17 01:24:51 mds1 kernel: 
Jan 17 01:25:33 mds1 kernel: LDISKFS-fs (dm-0): recovery complete
Jan 17 01:25:34 mds1 kernel: LDISKFS-fs (dm-1): recovery complete
Jan 17 01:25:34 mds1 kernel: LDISKFS-fs (dm-2): recovery complete
Jan 17 01:25:34 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:25:34 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:25:34 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:25:34 mds1 kernel: LustreError: 4165:0:(obd_mount_server.c:1123:server_register_target()) emc-MDT0000: error registering with the MGS: rc = -108 (not fatal)
Jan 17 01:25:34 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 17 01:25:34 mds1 kernel: LustreError: 4384:0:(sec_config.c:1121:sptlrpc_target_local_read_conf()) missing llog context
Jan 17 01:25:34 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:25:34 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 17 01:25:34 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:25:39 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389939934/real 0]  req@ffff8810088f4c00 x1457457563959316/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389939939 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:25:44 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389939939/real 0]  req@ffff8807f4fc5c00 x1457457563959536/t0(0) o250->MGC192.168.1.40@o2ib@192.168.1.42@o2ib:26/25 lens 400/544 e 0 to 1 dl 1389939944 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:25:44 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 17 01:25:44 mds1 kernel: LustreError: 4163:0:(ldlm_resource.c:809:ldlm_resource_complain()) MGC192.168.1.40@o2ib: namespace resource [0x636d65:0x0:0x0].0 (ffff8807f2e93080) refcount nonzero (1) after lock cleanup; forcing cleanup.
Jan 17 01:25:44 mds1 kernel: LustreError: 4163:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x636d65:0x0:0x0].0 (ffff8807f2e93080) refcount = 2
Jan 17 01:25:45 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 17 01:25:45 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_connect failed with -114.
Jan 17 01:25:45 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:25:45 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 2 clients reconnect
Jan 17 01:26:10 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:25, of 2 clients 2 recovered and 0 were evicted.
Jan 17 01:26:10 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 17 01:26:16 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389939970/real 0]  req@ffff8808309ca800 x1457457563960108/t0(0) o38->emc-MDT0000-osp-MDT0001@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389939976 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:26:35 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:50, of 2 clients 2 recovered and 0 were evicted.
Jan 17 01:26:35 mds1 kernel: Lustre: emc-MDT0000-osp-MDT0001: Connection restored to emc-MDT0000 (at 0@lo)
Jan 17 01:27:56 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 17 01:27:56 mds1 kernel: LustreError: 11-0: emc-MDT0000-osp-MDT0001: Communicating with 0@lo, operation mds_disconnect failed with -19.
Jan 17 01:27:56 mds1 kernel: LustreError: 5031:0:(osp_dev.c:338:osp_disconnect()) emc-MDT0000-osp-MDT0001: can't disconnect: rc = -19
Jan 17 01:27:56 mds1 kernel: LustreError: 5031:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031: rc = -19
Jan 17 01:27:56 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 17 01:28:06 mds1 kernel: Lustre: 5037:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389940076/real 1389940076]  req@ffff88100797b400 x1457457563964840/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389940086 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:28:06 mds1 kernel: Lustre: server umount MGS complete
Jan 17 01:28:06 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 01:29:50 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:29:52 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:29:52 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 17 01:29:52 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 17 01:29:52 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:29:52 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:29:52 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:30:17 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 01:30:17 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 17 01:30:17 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:30:18 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 2 clients reconnect
Jan 17 01:30:18 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 01:30:21 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
Jan 17 01:30:42 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 17 01:36:15 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:36:16 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 17 01:36:16 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:36:16 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:36:23 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389940567/real 0]  req@ffff880815d3b400 x1457457563972420/t0(0) o400->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1389940583 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:36:23 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 01:36:24 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 5:00, or until 2 clients reconnect
Jan 17 01:36:33 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389940583/real 1389940593]  req@ffff880832987800 x1457457563973120/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1389940594 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:36:48 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:24, of 2 clients 2 recovered and 0 were evicted.
Jan 17 01:36:48 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 17 01:39:19 mds1 kernel: Lustre: Failing over emc-MDT0001
Jan 17 01:39:19 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 01:39:19 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_disconnect failed with -19.
Jan 17 01:39:19 mds1 kernel: LustreError: 6777:0:(osp_dev.c:338:osp_disconnect()) emc-MDT0001-osp-MDT0000: can't disconnect: rc = -19
Jan 17 01:39:19 mds1 kernel: LustreError: 6777:0:(lod_dev.c:171:lod_cleanup_desc_tgts()) emc-MDT0000-mdtlov: error cleaning up LOD index 1: cmd 0xcf031: rc = -19
Jan 17 01:39:20 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 17 01:39:25 mds1 kernel: Lustre: 6776:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389940759/real 1389940759]  req@ffff881007679c00 x1457457563980208/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389940765 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:39:26 mds1 kernel: Lustre: server umount MGS complete
Jan 17 01:39:26 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 01:40:34 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:40:34 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 01:40:35 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 17 01:40:35 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -114.
Jan 17 01:40:35 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:40:35 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 01:40:35 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:41:00 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 01:41:00 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 17 01:41:00 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 01:41:10 mds1 kernel: Lustre: emc-MDT0000: Will be in recovery for at least 5:00, or until 2 clients reconnect
Jan 17 01:41:11 mds1 kernel: Lustre: emc-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
Jan 17 01:41:25 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 192.168.1.42@o2ib)
Jan 17 01:49:38 mds1 kernel: Lustre: 3360:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389941371/real 0]  req@ffff880feceec800 x1457457563990760/t0(0) o13->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389941378 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 01:49:38 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 01:49:38 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection to emc-OST0005 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 01:50:39 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389941428/real 0]  req@ffff880818181000 x1457457563991528/t0(0) o8->emc-OST0004-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389941439 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 01:50:39 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 17 01:50:40 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:50:40 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0008_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:50:41 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0009_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:50:41 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 01:50:42 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0001_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:50:42 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 01:50:47 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection restored to emc-OST0006 (at 192.168.1.46@o2ib)
Jan 17 01:51:32 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:51:32 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 01:52:22 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:52:22 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 01:52:47 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0001_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:52:47 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 17 01:53:12 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 01:53:16 mds1 kernel: Lustre: MGS: haven't heard from client f4c65f19-4f49-de52-34a3-6e466abfbbd0 (at 192.168.1.44@o2ib) in 235 seconds. I think it's dead, and I am evicting it. exp ffff880ff7a35400, cur 1389941596 expire 1389941446 last 1389941361
Jan 17 01:53:16 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0003_UUID (at 192.168.1.44@o2ib) in 235 seconds. I think it's dead, and I am evicting it. exp ffff8807f28bd800, cur 1389941596 expire 1389941446 last 1389941361
Jan 17 02:01:14 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389942067/real 0]  req@ffff881011a3c000 x1457457564004184/t0(0) o13->emc-OST0004-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389942074 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 02:01:14 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 17 02:01:14 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection to emc-OST0004 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:01:14 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 02:01:14 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:01:14 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 02:01:18 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:01:18 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 02:02:16 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0010_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 02:02:16 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 02:02:23 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection restored to emc-OST0010 (at 192.168.1.44@o2ib)
Jan 17 02:02:23 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 02:03:57 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0010_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 02:03:57 mds1 kernel: Lustre: Skipped 47 previous similar messages
Jan 17 02:04:37 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST000e_UUID (at 192.168.1.46@o2ib) in 235 seconds. I think it's dead, and I am evicting it. exp ffff8810118db400, cur 1389942277 expire 1389942127 last 1389942042
Jan 17 02:04:37 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 02:09:20 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389942548/real 0]  req@ffff8807fa0ae400 x1457457564012976/t0(0) o13->emc-OST0007-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389942560 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 02:09:20 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 96 previous similar messages
Jan 17 02:09:20 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection to emc-OST000f (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:09:20 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 02:10:20 mds1 kernel: Lustre: MGS: Client 35083410-077a-46e6-d4b3-41859cbe5f8a (at 192.168.1.46@o2ib) reconnecting
Jan 17 02:10:20 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0002_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 02:10:20 mds1 kernel: Lustre: Skipped 14 previous similar messages
Jan 17 02:10:29 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection restored to emc-OST0008 (at 192.168.1.46@o2ib)
Jan 17 02:10:29 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:12:52 mds1 kernel: Lustre: MGS: haven't heard from client 0595ba76-3ad1-9a89-0a06-01728fd2e7e8 (at 192.168.1.44@o2ib) in 235 seconds. I think it's dead, and I am evicting it. exp ffff88081592e400, cur 1389942772 expire 1389942622 last 1389942537
Jan 17 02:12:52 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 17 02:13:05 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0012_UUID (at 192.168.1.44@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff880824f88000, cur 1389942785 expire 1389942635 last 1389942537
Jan 17 02:20:51 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389943244/real 0]  req@ffff880ff7a59000 x1457457564025936/t0(0) o13->emc-OST000d-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389943251 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 02:20:51 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 95 previous similar messages
Jan 17 02:20:51 mds1 kernel: Lustre: emc-OST0016-osc-MDT0000: Connection to emc-OST0016 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:20:51 mds1 kernel: Lustre: Skipped 33 previous similar messages
Jan 17 02:21:57 mds1 kernel: Lustre: MGS: Client 4d2e13aa-206f-081b-34ee-21765f6f73e1 (at 192.168.1.44@o2ib) reconnecting
Jan 17 02:21:57 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0008_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 02:21:57 mds1 kernel: Lustre: Skipped 78 previous similar messages
Jan 17 02:22:04 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection restored to emc-OST0006 (at 192.168.1.44@o2ib)
Jan 17 02:22:04 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:22:06 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection restored to emc-OST0007 (at 192.168.1.44@o2ib)
Jan 17 02:22:06 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 17 02:22:31 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection restored to emc-OST0005 (at 192.168.1.44@o2ib)
Jan 17 02:22:31 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 17 02:24:11 mds1 kernel: Lustre: MGS: haven't heard from client 35083410-077a-46e6-d4b3-41859cbe5f8a (at 192.168.1.46@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880818797400, cur 1389943451 expire 1389943301 last 1389943222
Jan 17 02:24:11 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:28:51 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:28:51 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:28:51 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:28:51 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 17 02:28:53 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:28:53 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 17 02:29:29 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389943758/real 1389943758]  req@ffff8808193bc000 x1457457564034528/t0(0) o8->emc-OST0008-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389943769 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 02:29:29 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 141 previous similar messages
Jan 17 02:29:54 mds1 kernel: Lustre: MGS: Client 69c7a743-1c51-cd7c-9bc2-f1836aeee18e (at 192.168.1.46@o2ib) reconnecting
Jan 17 02:30:04 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection restored to emc-OST0015 (at 192.168.1.46@o2ib)
Jan 17 02:30:04 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 17 02:30:07 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection restored to emc-OST0003 (at 192.168.1.46@o2ib)
Jan 17 02:30:07 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 17 02:30:27 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 17 02:30:27 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 02:30:46 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0008_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 02:30:46 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0011_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 02:30:46 mds1 kernel: Lustre: Skipped 77 previous similar messages
Jan 17 02:32:32 mds1 kernel: Lustre: MGS: haven't heard from client 4d2e13aa-206f-081b-34ee-21765f6f73e1 (at 192.168.1.44@o2ib) in 233 seconds. I think it's dead, and I am evicting it. exp ffff880818bb5000, cur 1389943952 expire 1389943802 last 1389943719
Jan 17 02:32:32 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 17 02:32:37 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0008_UUID (at 192.168.1.44@o2ib) in 238 seconds. I think it's dead, and I am evicting it. exp ffff88082e688000, cur 1389943957 expire 1389943807 last 1389943719
Jan 17 02:40:26 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389944419/real 0]  req@ffff881030419000 x1457457564047040/t0(0) o13->emc-OST0013-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389944426 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 02:40:26 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 17 02:40:26 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection to emc-OST0013 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:40:26 mds1 kernel: Lustre: Skipped 15 previous similar messages
Jan 17 02:40:26 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection to emc-OST0015 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:40:26 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 17 02:40:29 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:40:29 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 02:41:33 mds1 kernel: Lustre: MGS: Client 34b475a8-dd72-5155-ed48-412c98c39265 (at 192.168.1.44@o2ib) reconnecting
Jan 17 02:41:33 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0011_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 02:41:33 mds1 kernel: Lustre: Skipped 58 previous similar messages
Jan 17 02:41:41 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 02:41:41 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 02:43:50 mds1 kernel: Lustre: MGS: haven't heard from client 69c7a743-1c51-cd7c-9bc2-f1836aeee18e (at 192.168.1.46@o2ib) in 234 seconds. I think it's dead, and I am evicting it. exp ffff88083034f000, cur 1389944630 expire 1389944480 last 1389944396
Jan 17 02:43:50 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:48:16 mds1 kernel: LustreError: 11-0: emc-OST0008-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 02:48:16 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:48:16 mds1 kernel: Lustre: Skipped 16 previous similar messages
Jan 17 02:48:16 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 02:48:28 mds1 kernel: Lustre: MGS: Client 44b91487-d3c8-a964-07d3-00a471e95f9a (at 192.168.1.46@o2ib) reconnecting
Jan 17 02:48:36 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection restored to emc-OST0003 (at 192.168.1.46@o2ib)
Jan 17 02:48:36 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 17 02:48:41 mds1 kernel: LustreError: 11-0: emc-OST0003-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 02:48:41 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:48:41 mds1 kernel: Lustre: Skipped 21 previous similar messages
Jan 17 02:48:41 mds1 kernel: LustreError: Skipped 25 previous similar messages
Jan 17 02:48:51 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection restored to emc-OST0003 (at 192.168.1.44@o2ib)
Jan 17 02:48:51 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 17 02:59:13 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389945546/real 0]  req@ffff880ff7ab2c00 x1457457564068860/t0(0) o13->emc-OST0011-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389945553 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 02:59:13 mds1 kernel: Lustre: 3359:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 95 previous similar messages
Jan 17 02:59:13 mds1 kernel: Lustre: emc-OST0011-osc-MDT0000: Connection to emc-OST0011 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 02:59:13 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 03:00:07 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0016_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 03:00:07 mds1 kernel: Lustre: Skipped 61 previous similar messages
Jan 17 03:00:24 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.44@o2ib)
Jan 17 03:00:24 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 03:00:49 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.44@o2ib)
Jan 17 03:00:49 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 03:02:58 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST000c_UUID (at 192.168.1.46@o2ib) in 244 seconds. I think it's dead, and I am evicting it. exp ffff8810123c7000, cur 1389945778 expire 1389945628 last 1389945534
Jan 17 03:02:58 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 17 03:07:01 mds1 kernel: LustreError: 11-0: emc-OST0005-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 03:07:01 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 17 03:07:01 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection to emc-OST0005 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 03:07:01 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 03:07:04 mds1 kernel: LustreError: 11-0: emc-OST000d-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 03:07:04 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 03:07:04 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 03:07:04 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 17 03:07:15 mds1 kernel: Lustre: MGS: Client bf4f5c45-ce38-1e78-459f-eb5fc978c544 (at 192.168.1.46@o2ib) reconnecting
Jan 17 03:07:24 mds1 kernel: Lustre: emc-OST0015-osc-MDT0000: Connection restored to emc-OST0015 (at 192.168.1.46@o2ib)
Jan 17 03:07:28 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 03:07:28 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 09:00:59 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 09:00:59 mds1 kernel: LustreError: Skipped 19 previous similar messages
Jan 17 09:00:59 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection to emc-OST0017 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:00:59 mds1 kernel: Lustre: Skipped 21 previous similar messages
Jan 17 09:01:18 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 09:01:18 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:01:19 mds1 kernel: LustreError: 11-0: emc-OST000b-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 09:01:19 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:01:19 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 09:01:19 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 17 09:01:31 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389967284/real 1389967284]  req@ffff880813172c00 x1457457564496512/t0(0) o13->emc-OST0012-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389967291 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 09:01:31 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
Jan 17 09:01:31 mds1 kernel: Lustre: emc-OST0012-osc-MDT0000: Connection to emc-OST0012 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:01:31 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 17 09:01:35 mds1 cibadmin[45336]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:01:35 mds1 cibadmin[45337]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:01:35 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 17 09:01:35 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 09:01:35 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 17 09:01:36 mds1 cibadmin[45414]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:01:36 mds1 cibadmin[45415]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:01:39 mds1 cibadmin[45419]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:01:39 mds1 cibadmin[45420]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:01:55 mds1 kernel: Lustre: server umount MGS complete
Jan 17 09:02:29 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:03:03 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:03:49 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:03:49 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:04:00 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:04:00 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:04:00 mds1 kernel: Lustre: ctl-emc-MDT0000: No data found on store. Initialize space
Jan 17 09:04:00 mds1 kernel: Lustre: emc-MDT0000: new disk, initializing
Jan 17 09:04:00 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 09:04:00 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 09:04:01 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 17 09:04:01 mds1 kernel: LustreError: Skipped 7 previous similar messages
Jan 17 09:04:23 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.50@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 09:04:52 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -11.
Jan 17 09:05:03 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 17 09:05:03 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 17 09:05:12 mds1 kernel: Lustre: 45870:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389967506/real 1389967506]  req@ffff880815731400 x1457457564496840/t0(0) o251->MGC192.168.1.40@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1389967512 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:05:12 mds1 kernel: Lustre: 45870:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 17 09:05:12 mds1 kernel: Lustre: server umount MGS complete
Jan 17 09:05:27 mds1 cibadmin[45899]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:05:27 mds1 cibadmin[45900]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:05:27 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:05:30 mds1 cibadmin[46058]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:05:30 mds1 cibadmin[46059]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:05:31 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 09:05:31 mds1 cibadmin[46142]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 09:05:31 mds1 cibadmin[46143]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 09:05:31 mds1 kernel: Lustre: emc-MDT0000: used disk, loading
Jan 17 09:05:31 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 09:05:31 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 09:05:31 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 17 09:05:56 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 09:05:56 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 17 09:06:21 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -11.
Jan 17 09:07:11 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -11.
Jan 17 09:07:11 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 09:08:26 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -11.
Jan 17 09:08:26 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 09:10:56 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 192.168.1.42@o2ib, operation mds_connect failed with -11.
Jan 17 09:10:56 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 17 09:12:18 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:ost
Jan 17 09:12:18 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400):e:ost
Jan 17 09:13:04 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000800000400-0x0000000840000400):0:mdt
Jan 17 09:13:04 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 17 09:34:01 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 09:34:01 mds1 kernel: LustreError: Skipped 3 previous similar messages
Jan 17 09:34:01 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:34:01 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 09:34:16 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection to emc-OST000b (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:34:21 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:34:21 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 17 09:34:55 mds1 cibadmin[48837]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 09:35:03 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969292/real 1389969292]  req@ffff8807fb2ac800 x1457457564532936/t0(0) o8->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969303 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:35:03 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 17 09:35:28 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969317/real 1389969317]  req@ffff8807f3e88800 x1457457564533276/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969328 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:36:23 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969367/real 1389969367]  req@ffff880803a3c800 x1457457564533876/t0(0) o8->emc-OST0006-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:36:23 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 17 09:37:48 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969442/real 1389969442]  req@ffff880802403400 x1457457564534736/t0(0) o8->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:37:48 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Jan 17 09:38:21 mds1 kernel: LustreError: 11-0: emc-OST000e-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 09:38:21 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 17 09:38:21 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 09:38:21 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 09:40:48 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969617/real 1389969617]  req@ffff880830914000 x1457457564535908/t0(0) o8->emc-OST0002-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969648 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:40:48 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 17 09:41:32 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client aa1da60c-84ee-d309-3f6e-3847e23c9f6b (at 192.168.1.56@o2ib) in 232 seconds. I think it's dead, and I am evicting it. exp ffff88100166ec00, cur 1389969692 expire 1389969542 last 1389969460
Jan 17 09:41:32 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 17 09:45:53 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389969917/real 1389969917]  req@ffff8808004ac800 x1457457564536856/t0(0) o8->emc-OST0015-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389969953 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:45:53 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 131 previous similar messages
Jan 17 09:56:12 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389970517/real 1389970517]  req@ffff8807f7936400 x1457457564537972/t0(0) o8->emc-OST0002-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389970572 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 09:56:12 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 211 previous similar messages
Jan 17 10:00:57 mds1 cibadmin[51075]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:00:57 mds1 cibadmin[51076]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:00:58 mds1 kernel: Lustre: server umount MGS complete
Jan 17 10:01:00 mds1 cibadmin[51153]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:01:00 mds1 cibadmin[51154]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:01:00 mds1 kernel: Lustre: Failing over emc-MDT0000
Jan 17 10:01:01 mds1 cibadmin[51215]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:01:01 mds1 cibadmin[51216]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:01:03 mds1 kernel: LustreError: 137-5: emc-MDT0000_UUID: not available for connect from 192.168.1.42@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 10:01:03 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 10:01:06 mds1 kernel: Lustre: server umount emc-MDT0000 complete
Jan 17 10:02:32 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:02:41 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:02:50 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:03:01 mds1 cibadmin[51394]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:03:01 mds1 cibadmin[51395]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:03:02 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:03:02 mds1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:03:04 mds1 cibadmin[51567]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:03:04 mds1 cibadmin[51568]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:03:05 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:03:06 mds1 cibadmin[51684]:   notice: crm_log_args: Invoked: cibadmin -Ql -o configuration 
Jan 17 10:03:06 mds1 cibadmin[51685]:   notice: crm_log_args: Invoked: cibadmin -p -R -o resources 
Jan 17 10:03:06 mds1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 17 10:03:06 mds1 kernel: Lustre: ctl-emc-MDT0000: No data found on store. Initialize space
Jan 17 10:03:06 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 10:03:06 mds1 kernel: Lustre: emc-MDT0000: new disk, initializing
Jan 17 10:03:06 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 17 10:03:06 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 10:03:06 mds1 kernel: LustreError: 11-0: emc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Jan 17 10:03:06 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 17 10:03:33 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 10:03:33 mds1 kernel: LustreError: 11-0: emc-MDT0001-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -19.
Jan 17 10:03:33 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 10:03:50 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:ost
Jan 17 10:04:12 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:04:12 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 10:05:38 mds1 kernel: Lustre: Denying initial registration attempt from nid 192.168.1.46@o2ib, specified as failover
Jan 17 10:05:39 mds1 kernel: Lustre: Denying initial registration attempt from nid 192.168.1.46@o2ib, specified as failover
Jan 17 10:05:39 mds1 kernel: Lustre: Denying initial registration attempt from nid 192.168.1.46@o2ib, specified as failover
Jan 17 10:05:39 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 17 10:05:44 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.46@o2ib)
Jan 17 10:05:44 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 10:05:44 mds1 kernel: Lustre: Denying initial registration attempt from nid 192.168.1.46@o2ib, specified as failover
Jan 17 10:05:44 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 10:05:54 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400):c:ost
Jan 17 10:06:06 mds1 kernel: LustreError: 11-0: emc-OST000c-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 10:06:06 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection to emc-OST000c (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:06:17 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.44@o2ib)
Jan 17 10:06:29 mds1 kernel: Lustre: 3362:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389971182/real 1389971182]  req@ffff880822225000 x1457457564539312/t0(0) o13->emc-OST000c-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389971189 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 10:06:29 mds1 kernel: Lustre: 3362:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 101 previous similar messages
Jan 17 10:06:29 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection to emc-OST000c (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:07:30 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 17 10:07:59 mds1 kernel: LustreError: 13b-9: emc-OST000e claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:08:00 mds1 kernel: LustreError: 13b-9: emc-OST0017 claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:08:00 mds1 kernel: LustreError: 13b-9: emc-OST0012 claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:08:00 mds1 kernel: LustreError: Skipped 7 previous similar messages
Jan 17 10:08:05 mds1 kernel: LustreError: 11-0: emc-OST000c-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 10:08:05 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection to emc-OST000c (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:08:27 mds1 kernel: Lustre: Denying initial registration attempt from nid 192.168.1.46@o2ib, specified as failover
Jan 17 10:08:27 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 10:09:21 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400):1:ost
Jan 17 10:09:29 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x00000004c0000400-0x0000000500000400):a:ost
Jan 17 10:09:29 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 17 10:10:00 mds1 kernel: LustreError: 13b-9: emc-OST000d claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:10:00 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 10:10:03 mds1 kernel: LustreError: 13b-9: emc-OST0010 claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:10:03 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 10:10:08 mds1 kernel: LustreError: 13b-9: emc-OST0015 claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:10:08 mds1 kernel: LustreError: Skipped 4 previous similar messages
Jan 17 10:10:08 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 17 10:10:24 mds1 kernel: LustreError: 13b-9: emc-OST000d claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:10:24 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 10:11:03 mds1 kernel: LustreError: 13b-9: emc-OST000d claims to have registered, but this MGS does not know about it, preventing registration.
Jan 17 10:11:03 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 17 10:13:19 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 10:13:19 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:13:33 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 10:14:05 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:17:30 mds1 kernel: LustreError: 11-0: emc-OST0002-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 10:17:30 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 17 10:17:30 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:24:16 mds1 kernel: Lustre: 3360:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389972249/real 0]  req@ffff88103269bc00 x1457457564549276/t0(0) o13->emc-OST0008-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389972256 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 10:24:16 mds1 kernel: Lustre: 3360:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jan 17 10:24:16 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection to emc-OST0008 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:25:36 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0000_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 10:25:36 mds1 kernel: Lustre: Skipped 43 previous similar messages
Jan 17 10:25:38 mds1 kernel: LustreError: 11-0: emc-OST000c-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 10:25:38 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection to emc-OST000c (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:25:38 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 17 10:25:45 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.46@o2ib)
Jan 17 10:26:20 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.46@o2ib)
Jan 17 10:27:18 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0000_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 10:27:18 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 10:27:19 mds1 kernel: LustreError: 51785:0:(ldlm_lockd.c:662:ldlm_handle_ast_error()) ### client (nid 192.168.1.44@o2ib) returned 0 from blocking AST ns: MGS lock: ffff880821c227c0/0x26009b5cfd834ab5 lrc: 4/0,0 mode: CR/CR res: [0x636d65:0x0:0x0].0 rrc: 33 type: PLN flags: 0x60000000000020 nid: 192.168.1.44@o2ib remote: 0x563b85a3332a14d expref: 53 pid: 51785 timeout: 4327441819 lvb_type: 0
Jan 17 10:27:19 mds1 kernel: LustreError: 51785:0:(ldlm_lockd.c:662:ldlm_handle_ast_error()) ### client (nid 192.168.1.44@o2ib) returned 0 from blocking AST ns: MGS lock: ffff881001bc88c0/0x26009b5cfd83488c lrc: 4/0,0 mode: CR/CR res: [0x636d65:0x0:0x0].0 rrc: 10 type: PLN flags: 0x60000000000020 nid: 192.168.1.44@o2ib remote: 0x563b85a33329fb7 expref: 37 pid: 51785 timeout: 4327441819 lvb_type: 0
Jan 17 10:27:19 mds1 kernel: LustreError: 51785:0:(ldlm_lockd.c:662:ldlm_handle_ast_error()) Skipped 15 previous similar messages
Jan 17 10:27:22 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000540000400-0x0000000580000400):17:ost
Jan 17 10:27:22 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 10:27:46 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0003_UUID (at 192.168.1.44@o2ib) in 238 seconds. I think it's dead, and I am evicting it. exp ffff880832598800, cur 1389972466 expire 1389972316 last 1389972228
Jan 17 10:27:46 mds1 kernel: Lustre: emc-MDT0000: haven't heard from client emc-MDT0000-lwp-OST0006_UUID (at 192.168.1.44@o2ib) in 238 seconds. I think it's dead, and I am evicting it. exp ffff880814d3c000, cur 1389972466 expire 1389972316 last 1389972228
Jan 17 10:27:47 mds1 kernel: Lustre: MGS: haven't heard from client 68eae4b0-80e4-7a1e-dfab-4af250e52a44 (at 192.168.1.44@o2ib) in 239 seconds. I think it's dead, and I am evicting it. exp ffff881012150400, cur 1389972467 expire 1389972317 last 1389972228
Jan 17 10:27:47 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 17 10:27:47 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 17 10:28:25 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 17 10:34:30 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389972870/real 1389972870]  req@ffff8807fb08f800 x1457457564556540/t0(0) o8->emc-OST0003-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389972925 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 10:34:30 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 152 previous similar messages
Jan 17 10:36:17 mds1 kernel: Lustre: emc-OST0010-osc-MDT0000: Connection to emc-OST0010 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:37:18 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0000_UUID seen on new nid 192.168.1.44@o2ib when existing nid 192.168.1.46@o2ib is already connected
Jan 17 10:37:28 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 10:39:03 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 10:39:03 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:39:03 mds1 kernel: Lustre: Skipped 14 previous similar messages
Jan 17 10:40:04 mds1 kernel: Lustre: MGS: haven't heard from client 80f1eb40-c4a0-2000-63b1-8a80ff252e4e (at 192.168.1.46@o2ib) in 238 seconds. I think it's dead, and I am evicting it. exp ffff881011ee9400, cur 1389973204 expire 1389973054 last 1389972966
Jan 17 10:43:24 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 10:44:33 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1389973449/real 1389973473]  req@ffff880824f5c000 x1457457564559468/t0(0) o13->emc-OST0000-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389973476 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 10:44:33 mds1 kernel: Lustre: 3357:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 250 previous similar messages
Jan 17 10:44:33 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:45:05 mds1 cibadmin[55531]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 10:45:40 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0000_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 17 10:45:40 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 10:45:53 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 17 10:47:55 mds1 kernel: Lustre: MGS: haven't heard from client 4934914b-cde8-bfe8-25ba-4e82403d18a7 (at 192.168.1.44@o2ib) in 231 seconds. I think it's dead, and I am evicting it. exp ffff8807fd024c00, cur 1389973675 expire 1389973525 last 1389973444
Jan 17 10:47:55 mds1 kernel: Lustre: Skipped 15 previous similar messages
Jan 17 10:54:54 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389974048/real 0]  req@ffff880824c10800 x1457457564562268/t0(0) o8->emc-OST000b-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389974094 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 10:54:54 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 175 previous similar messages
Jan 17 10:56:05 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 10:56:05 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 10:57:05 mds1 kernel: Lustre: MGS: Client df959924-c7f9-4f9a-c8bd-04217af25805 (at 192.168.1.44@o2ib) reconnecting
Jan 17 10:57:08 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 10:57:08 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 11:00:01 mds1 kernel: Lustre: emc-OST0003-osc-MDT0000: Connection restored to emc-OST0003 (at 192.168.1.46@o2ib)
Jan 17 11:02:02 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection restored to emc-OST0007 (at 192.168.1.46@o2ib)
Jan 17 11:02:02 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 17 11:03:19 mds1 kernel: Lustre: emc-OST0004-osc-MDT0000: Connection restored to emc-OST0004 (at 192.168.1.44@o2ib)
Jan 17 11:03:19 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 11:05:03 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389974648/real 1389974648]  req@ffff880821c41000 x1457457564566352/t0(0) o8->emc-OST0012-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389974703 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 11:05:03 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 41 previous similar messages
Jan 17 11:05:53 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection restored to emc-OST0008 (at 192.168.1.44@o2ib)
Jan 17 11:05:53 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 17 11:09:53 mds1 kernel: Lustre: emc-OST000d-osc-MDT0000: Connection restored to emc-OST000d (at 192.168.1.46@o2ib)
Jan 17 11:09:53 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 17 11:13:15 mds1 kernel: Lustre: ctl-emc-MDT0000: super-sequence allocation rc = 0 [0x0000000800000400-0x0000000840000400):0:mdt
Jan 17 11:13:15 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 12:44:22 mds1 kernel: LustreError: 11-0: emc-OST0002-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 12:44:22 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 12:44:22 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 12:44:22 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 17 12:44:30 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection restored to emc-OST0006 (at 192.168.1.46@o2ib)
Jan 17 12:44:30 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 12:59:00 mds1 cibadmin[67186]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 13:00:30 mds1 kernel: LustreError: 11-0: emc-OST0001-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 13:00:30 mds1 kernel: LustreError: Skipped 4 previous similar messages
Jan 17 13:00:30 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 13:00:30 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 13:00:40 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.44@o2ib)
Jan 17 13:00:40 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 13:00:58 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 13:00:58 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 13:00:58 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 13:00:58 mds1 kernel: LustreError: Skipped 15 previous similar messages
Jan 17 13:01:04 mds1 kernel: Lustre: emc-OST000d-osc-MDT0000: Connection restored to emc-OST000d (at 192.168.1.44@o2ib)
Jan 17 13:01:04 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 13:01:06 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389981658/real 1389981658]  req@ffff8807f4c12000 x1457457564739916/t0(0) o13->emc-OST000e-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389981666 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 13:01:06 mds1 kernel: Lustre: 3353:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 42 previous similar messages
Jan 17 13:01:13 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.44@o2ib)
Jan 17 13:01:13 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 13:17:50 mds1 kernel: LustreError: 11-0: emc-OST0001-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 13:17:50 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 17 13:17:50 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection to emc-OST0001 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 13:17:50 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 13:17:53 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 13:17:53 mds1 kernel: LustreError: Skipped 2 previous similar messages
Jan 17 13:17:53 mds1 kernel: Lustre: emc-OST0006-osc-MDT0000: Connection to emc-OST0006 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 13:17:53 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 13:18:01 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.46@o2ib)
Jan 17 13:18:06 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.46@o2ib)
Jan 17 13:18:06 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 13:18:33 mds1 kernel: LustreError: 11-0: emc-OST000e-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 13:18:33 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 17 13:18:33 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 13:18:33 mds1 kernel: Lustre: Skipped 8 previous similar messages
Jan 17 13:18:43 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection restored to emc-OST0013 (at 192.168.1.46@o2ib)
Jan 17 13:18:43 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 13:36:17 mds1 cibadmin[70411]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 14:08:19 mds1 kernel: Lustre: MGS: Client b8bfdaa0-32cd-48e5-3f92-4065d8ac0b4f (at 192.168.1.54@o2ib) reconnecting
Jan 17 14:08:19 mds1 kernel: LustreError: 137-5: emc-MDT0001_UUID: not available for connect from 192.168.1.54@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jan 17 14:08:26 mds1 kernel: Lustre: emc-MDT0000: Client 748c5d38-ff83-5478-200a-f3bc1aac8e26 (at 192.168.1.54@o2ib) reconnecting
Jan 17 14:08:26 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 14:15:46 mds1 kernel: LustreError: 11-0: emc-OST0002-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 14:15:46 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 17 14:15:46 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:15:46 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 14:15:51 mds1 kernel: LustreError: 11-0: emc-OST000b-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 14:15:51 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 17 14:15:51 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection to emc-OST000b (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:15:51 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 14:15:54 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection restored to emc-OST0009 (at 192.168.1.44@o2ib)
Jan 17 14:15:54 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 14:16:03 mds1 kernel: LustreError: 11-0: emc-OST000e-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 14:16:03 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:16:09 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection to emc-OST0014 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:16:09 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 17 14:16:12 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.44@o2ib)
Jan 17 14:16:12 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 14:20:41 mds1 cibadmin[74256]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 14:26:34 mds1 cibadmin[74760]:   notice: crm_log_args: Invoked: cibadmin -Ql 
Jan 17 14:36:19 mds1 kernel: LustreError: 11-0: emc-OST0009-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 14:36:19 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection to emc-OST0002 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:36:19 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 17 14:36:19 mds1 kernel: LustreError: Skipped 4 previous similar messages
Jan 17 14:36:22 mds1 kernel: LustreError: 11-0: emc-OST000d-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 14:36:22 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection to emc-OST000e (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:36:22 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 14:36:22 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 17 14:37:10 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389987423/real 1389987423]  req@ffff880fec477400 x1457457564863568/t0(0) o8->emc-OST0012-osc-MDT0000@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389987430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 14:37:10 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 17 14:37:10 mds1 kernel: LustreError: 167-0: emc-OST0001-osc-MDT0000: This client was evicted by emc-OST0001; in progress operations using this service will fail.
Jan 17 14:37:10 mds1 kernel: Lustre: emc-OST0001-osc-MDT0000: Connection restored to emc-OST0001 (at 192.168.1.44@o2ib)
Jan 17 14:37:10 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 14:37:10 mds1 kernel: LustreError: 167-0: emc-OST0004-osc-MDT0000: This client was evicted by emc-OST0004; in progress operations using this service will fail.
Jan 17 14:37:10 mds1 kernel: LustreError: Skipped 5 previous similar messages
Jan 17 14:37:18 mds1 kernel: LustreError: 167-0: emc-OST0008-osc-MDT0000: This client was evicted by emc-OST0008; in progress operations using this service will fail.
Jan 17 14:37:18 mds1 kernel: Lustre: emc-OST0008-osc-MDT0000: Connection restored to emc-OST0008 (at 192.168.1.44@o2ib)
Jan 17 14:37:18 mds1 kernel: Lustre: Skipped 6 previous similar messages
Jan 17 14:37:28 mds1 kernel: LustreError: 167-0: emc-OST0013-osc-MDT0000: This client was evicted by emc-OST0013; in progress operations using this service will fail.
Jan 17 14:37:28 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 17 14:37:28 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection restored to emc-OST0013 (at 192.168.1.44@o2ib)
Jan 17 14:37:28 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 17 14:37:35 mds1 kernel: LustreError: 167-0: emc-OST0000-osc-MDT0000: This client was evicted by emc-OST0000; in progress operations using this service will fail.
Jan 17 14:37:35 mds1 kernel: LustreError: Skipped 4 previous similar messages
Jan 17 14:37:35 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 17 14:37:35 mds1 kernel: Lustre: Skipped 4 previous similar messages
Jan 17 14:44:13 mds1 kernel: LustreError: 11-0: emc-OST0017-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 14:44:13 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 17 14:44:13 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection to emc-OST0017 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:44:13 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 14:44:36 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 17 14:45:24 mds1 kernel: Lustre: 3358:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389987916/real 1389987916]  req@ffff8807f6292400 x1457457564875844/t0(0) o13->emc-OST0017-osc-MDT0000@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389987924 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 14:45:24 mds1 kernel: Lustre: 3358:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Jan 17 14:45:24 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection to emc-OST0017 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 14:45:37 mds1 kernel: LustreError: 167-0: emc-OST0017-osc-MDT0000: This client was evicted by emc-OST0017; in progress operations using this service will fail.
Jan 17 14:45:37 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.44@o2ib)
Jan 17 15:27:40 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 15:27:40 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 15:27:43 mds1 kernel: LustreError: 11-0: emc-OST0008-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 17 15:27:43 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection to emc-OST0009 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 15:27:43 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 17 15:27:43 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 17 15:27:54 mds1 kernel: Lustre: 3354:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389990467/real 1389990467]  req@ffff8807f62b7400 x1457457564926096/t0(0) o13->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1389990474 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 17 15:27:54 mds1 kernel: Lustre: 3354:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 17 15:27:54 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection to emc-OST0017 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 15:27:54 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 17 15:28:00 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1389990474/real 1389990474]  req@ffff8807f3367400 x1457457564926132/t0(0) o8->emc-OST0017-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389990480 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 15:28:00 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jan 17 15:28:29 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection restored to emc-OST0007 (at 192.168.1.46@o2ib)
Jan 17 15:28:56 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389990525/real 0]  req@ffff8807f59e7c00 x1457457564926732/t0(0) o8->emc-OST0012-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389990536 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 15:28:56 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jan 17 15:29:17 mds1 kernel: Lustre: emc-OST000c-osc-MDT0000: Connection restored to emc-OST000c (at 192.168.1.46@o2ib)
Jan 17 15:29:17 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 17 15:29:51 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1389990575/real 0]  req@ffff8807f5887400 x1457457564927388/t0(0) o8->emc-OST0012-osc-MDT0000@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1389990591 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 17 15:29:51 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 17 15:29:51 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection restored to emc-OST0013 (at 192.168.1.46@o2ib)
Jan 17 15:30:16 mds1 kernel: Lustre: emc-OST0017-osc-MDT0000: Connection restored to emc-OST0017 (at 192.168.1.46@o2ib)
Jan 17 15:30:16 mds1 kernel: Lustre: Skipped 9 previous similar messages
Jan 17 15:39:14 mds1 kernel: LustreError: 11-0: emc-OST0007-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 17 15:39:14 mds1 kernel: LustreError: Skipped 8 previous similar messages
Jan 17 15:39:14 mds1 kernel: Lustre: emc-OST0007-osc-MDT0000: Connection to emc-OST0007 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 17 15:39:14 mds1 kernel: Lustre: Skipped 5 previous similar messages
Jan 17 15:39:19 mds1 kernel: Lustre: MGS: Client 222cf39c-9c21-35c5-47fa-15205e229374 (at 192.168.1.44@o2ib) reconnecting
Jan 17 15:39:19 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 17 15:39:29 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 18 17:51:49 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 18 17:51:49 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 18 17:51:49 mds1 kernel: device-mapper: table: 253:3: multipath: error getting device
Jan 18 17:51:49 mds1 kernel: device-mapper: ioctl: error adding target to table
Jan 18 18:05:04 mds1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 18 18:05:05 mds1 kernel: Lustre: emc-MDT0001: used disk, loading
Jan 18 18:05:05 mds1 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy.
Jan 18 18:05:05 mds1 kernel: LustreError: Skipped 1 previous similar message
Jan 18 18:05:05 mds1 kernel: Lustre: emc-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jan 18 18:05:10 mds1 kernel: Lustre: emc-MDT0001: Will be in recovery for at least 2:30, or until 6 clients reconnect
Jan 18 18:05:37 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390086330/real 1390086330]  req@ffff8807f73e8800 x1457457566814820/t0(0) o400->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 224/224 e 0 to 1 dl 1390086337 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 18 18:05:37 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Jan 18 18:05:37 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection to emc-MDT0001 (at 192.168.1.42@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:05:37 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 18 18:05:43 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390086337/real 1390086337]  req@ffff8807f5ced800 x1457457566815072/t0(0) o38->emc-MDT0001-osp-MDT0000@192.168.1.42@o2ib:24/10 lens 400/544 e 0 to 1 dl 1390086343 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 18 18:06:02 mds1 kernel: Lustre: emc-MDT0001: Recovery over after 0:52, of 6 clients 6 recovered and 0 were evicted.
Jan 18 18:06:02 mds1 kernel: Lustre: emc-MDT0001-osp-MDT0000: Connection restored to emc-MDT0001 (at 0@lo)
Jan 18 18:06:02 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 18 18:13:12 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 748c5d38-ff83-5478-200a-f3bc1aac8e26 (at 192.168.1.54@o2ib) in 232 seconds. I think it's dead, and I am evicting it. exp ffff880ff0d96400, cur 1390086792 expire 1390086642 last 1390086560
Jan 18 18:13:12 mds1 kernel: Lustre: Skipped 1 previous similar message
Jan 18 18:38:27 mds1 kernel: LustreError: 11-0: emc-OST0005-osc-MDT0001: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 18 18:38:27 mds1 kernel: LustreError: Skipped 11 previous similar messages
Jan 18 18:38:27 mds1 kernel: Lustre: emc-OST0005-osc-MDT0001: Connection to emc-OST0005 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:38:27 mds1 kernel: LustreError: 11-0: emc-OST000a-osc-MDT0001: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 18 18:38:27 mds1 kernel: LustreError: Skipped 10 previous similar messages
Jan 18 18:38:27 mds1 kernel: Lustre: emc-OST000a-osc-MDT0001: Connection to emc-OST000a (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:38:27 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 18 18:38:29 mds1 kernel: LustreError: 11-0: emc-OST0000-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 18 18:38:29 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection to emc-OST0000 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:38:40 mds1 kernel: Lustre: emc-OST0009-osc-MDT0000: Connection restored to emc-OST0009 (at 192.168.1.46@o2ib)
Jan 18 18:49:40 mds1 kernel: LustreError: 11-0: emc-OST0004-osc-MDT0001: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 18 18:49:40 mds1 kernel: Lustre: emc-OST0000-osc-MDT0001: Connection to emc-OST0000 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:49:40 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 18 18:49:40 mds1 kernel: LustreError: Skipped 23 previous similar messages
Jan 18 18:49:51 mds1 kernel: Lustre: emc-OST000b-osc-MDT0000: Connection restored to emc-OST000b (at 192.168.1.44@o2ib)
Jan 18 18:49:51 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 18 18:49:56 mds1 kernel: LustreError: 11-0: emc-OST0006-osc-MDT0000: Communicating with 192.168.1.44@o2ib, operation ost_statfs failed with -107.
Jan 18 18:49:56 mds1 kernel: Lustre: emc-OST000b-osc-MDT0001: Connection to emc-OST000b (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:49:56 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 18 18:49:56 mds1 kernel: LustreError: Skipped 21 previous similar messages
Jan 18 18:50:46 mds1 kernel: Lustre: emc-OST0000-osc-MDT0000: Connection restored to emc-OST0000 (at 192.168.1.44@o2ib)
Jan 18 18:50:46 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 18 18:51:31 mds1 kernel: LustreError: 11-0: emc-OST0011-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 18 18:51:31 mds1 kernel: Lustre: emc-OST0013-osc-MDT0000: Connection to emc-OST0013 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:51:31 mds1 kernel: Lustre: Skipped 22 previous similar messages
Jan 18 18:51:31 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 18 18:51:39 mds1 kernel: Lustre: 3356:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390089092/real 1390089092]  req@ffff8807f8449000 x1457457566928900/t0(0) o13->emc-OST000e-osc-MDT0001@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1390089099 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 18 18:51:39 mds1 kernel: Lustre: 3356:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 18 18:51:39 mds1 kernel: Lustre: emc-OST0013-osc-MDT0001: Connection to emc-OST0013 (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:51:39 mds1 kernel: Lustre: Skipped 12 previous similar messages
Jan 18 18:51:39 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390089092/real 1390089092]  req@ffff880ff1772800 x1457457566928920/t0(0) o13->emc-OST0015-osc-MDT0001@192.168.1.46@o2ib:7/4 lens 224/368 e 0 to 1 dl 1390089099 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jan 18 18:51:39 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jan 18 18:51:45 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390089099/real 1390089099]  req@ffff8807f6c90c00 x1457457566929104/t0(0) o8->emc-OST0012-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1390089105 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 18 18:51:45 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jan 18 18:51:45 mds1 kernel: Lustre: emc-OST0017-osc-MDT0001: Connection restored to emc-OST0017 (at 192.168.1.44@o2ib)
Jan 18 18:51:45 mds1 kernel: Lustre: Skipped 23 previous similar messages
Jan 18 18:52:10 mds1 kernel: Lustre: emc-OST000c-osc-MDT0001: Connection restored to emc-OST000c (at 192.168.1.44@o2ib)
Jan 18 18:52:10 mds1 kernel: Lustre: Skipped 19 previous similar messages
Jan 18 18:59:23 mds1 kernel: Lustre: 3360:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1390089556/real 0]  req@ffff880fe91b3800 x1457457566949732/t0(0) o13->emc-OST0003-osc-MDT0001@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1390089563 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 18 18:59:23 mds1 kernel: Lustre: 3360:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jan 18 18:59:23 mds1 kernel: Lustre: emc-OST0003-osc-MDT0001: Connection to emc-OST0003 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:59:23 mds1 kernel: Lustre: Skipped 10 previous similar messages
Jan 18 18:59:27 mds1 kernel: Lustre: emc-OST0013-osc-MDT0001: Connection to emc-OST0013 (at 192.168.1.44@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 18:59:27 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 18 18:59:28 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1390089560/real 0]  req@ffff8808250b6800 x1457457566949784/t0(0) o13->emc-OST000e-osc-MDT0000@192.168.1.44@o2ib:7/4 lens 224/368 e 0 to 1 dl 1390089568 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jan 18 18:59:28 mds1 kernel: Lustre: 3361:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 33 previous similar messages
Jan 18 18:59:59 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1390089593/real 1390089593]  req@ffff8807f68a4c00 x1457457566950232/t0(0) o8->emc-OST0011-osc-MDT0001@192.168.1.46@o2ib:28/4 lens 400/544 e 0 to 1 dl 1390089599 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 18 18:59:59 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 61 previous similar messages
Jan 18 19:00:27 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST000c_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 18 19:00:27 mds1 kernel: Lustre: Skipped 2 previous similar messages
Jan 18 19:00:29 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1390089618/real 0]  req@ffff8807f64a2000 x1457457566950464/t0(0) o8->emc-OST0012-osc-MDT0001@192.168.1.44@o2ib:28/4 lens 400/544 e 0 to 1 dl 1390089629 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jan 18 19:00:29 mds1 kernel: Lustre: 3352:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jan 18 19:01:45 mds1 kernel: Lustre: emc-MDT0000: Client emc-MDT0000-lwp-OST0007_UUID seen on new nid 192.168.1.46@o2ib when existing nid 192.168.1.44@o2ib is already connected
Jan 18 19:01:45 mds1 kernel: Lustre: Skipped 24 previous similar messages
Jan 18 19:02:02 mds1 kernel: Lustre: emc-MDT0001: haven't heard from client 2fc8cfa0-6f58-7ca9-70d4-971b88571da4 (at 192.168.1.55@o2ib) in 246 seconds. I think it's dead, and I am evicting it. exp ffff88082f0e8400, cur 1390089722 expire 1390089572 last 1390089476
Jan 18 19:02:02 mds1 kernel: Lustre: Skipped 7 previous similar messages
Jan 18 19:03:00 mds1 kernel: Lustre: MGS: haven't heard from client 3680f2c7-707d-9675-ec22-c22e2ddbbf0d (at 192.168.1.44@o2ib) in 234 seconds. I think it's dead, and I am evicting it. exp ffff881011e0d000, cur 1390089780 expire 1390089630 last 1390089546
Jan 18 19:03:00 mds1 kernel: Lustre: Skipped 27 previous similar messages
Jan 18 19:03:12 mds1 kernel: Lustre: emc-OST000f-osc-MDT0000: Connection restored to emc-OST000f (at 192.168.1.46@o2ib)
Jan 18 19:03:12 mds1 kernel: Lustre: Skipped 3 previous similar messages
Jan 18 19:03:12 mds1 kernel: Lustre: emc-OST000e-osc-MDT0000: Connection restored to emc-OST000e (at 192.168.1.46@o2ib)
Jan 18 19:03:12 mds1 kernel: Lustre: Skipped 35 previous similar messages
Jan 18 19:11:02 mds1 kernel: LustreError: 11-0: emc-OST000f-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 18 19:11:02 mds1 kernel: Lustre: emc-OST000b-osc-MDT0001: Connection to emc-OST000b (at 192.168.1.46@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 18 19:11:02 mds1 kernel: Lustre: Skipped 21 previous similar messages
Jan 18 19:11:02 mds1 kernel: LustreError: Skipped 12 previous similar messages
Jan 18 19:11:02 mds1 kernel: LustreError: 11-0: emc-OST000e-osc-MDT0000: Communicating with 192.168.1.46@o2ib, operation ost_statfs failed with -107.
Jan 18 19:11:02 mds1 kernel: LustreError: Skipped 38 previous similar messages
Jan 18 19:11:14 mds1 kernel: Lustre: MGS: Client b20093ab-8017-4855-2dcc-8d38802b236d (at 192.168.1.44@o2ib) reconnecting
Jan 18 19:11:27 mds1 kernel: Lustre: emc-OST0005-osc-MDT0000: Connection restored to emc-OST0005 (at 192.168.1.44@o2ib)
Jan 18 19:11:27 mds1 kernel: Lustre: Skipped 11 previous similar messages
Jan 18 19:11:31 mds1 kernel: Lustre: emc-OST0014-osc-MDT0000: Connection restored to emc-OST0014 (at 192.168.1.46@o2ib)
Jan 18 19:11:31 mds1 kernel: Lustre: Skipped 21 previous similar messages
Jan 18 19:11:52 mds1 kernel: Lustre: emc-OST0002-osc-MDT0000: Connection restored to emc-OST0002 (at 192.168.1.44@o2ib)
Jan 18 19:11:52 mds1 kernel: Lustre: Skipped 23 previous similar messages
